This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new afae3f2881 [Fix][Connector-V2][Iceberg] Bound sink writer memory with
flush_row_size (#12937)
afae3f2881 is described below
commit afae3f28810fdf8c9fae8d4121f967e73eb2f2f5
Author: zhengxiang378928908-code <[email protected]>
AuthorDate: Tue Nov 17 15:56:23 2026 +0000
[Fix][Connector-V2][Iceberg] Bound sink writer memory with flush_row_size
(#12937)
Co-authored-by: 老王 <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
docs/en/connectors/sink/Iceberg.md | 22 ++
.../introduction/concepts/incompatible-changes.md | 8 +
docs/zh/connectors/sink/Iceberg.md | 22 ++
.../introduction/concepts/incompatible-changes.md | 8 +
.../iceberg/config/IcebergSinkConfig.java | 9 +
.../iceberg/config/IcebergSinkOptions.java | 10 +
.../seatunnel/iceberg/sink/IcebergSinkFactory.java | 3 +-
.../sink/commit/IcebergAggregatedCommitter.java | 116 +++++-
.../iceberg/sink/commit/IcebergFilesCommitter.java | 108 ++++-
.../iceberg/sink/writer/IcebergRecordWriter.java | 12 +
.../iceberg/sink/writer/RecordWriter.java | 5 +
.../iceberg/config/IcebergSinkConfigTest.java | 33 ++
.../sink/commit/IcebergWriterRotationTest.java | 437 +++++++++++++++++++++
.../e2e/connector/iceberg/IcebergSinkIT.java | 108 +++++
.../fake_to_iceberg_flush_row_size_cdc.conf | 65 +++
.../fake_to_iceberg_flush_row_size_upsert.conf | 66 ++++
16 files changed, 1010 insertions(+), 22 deletions(-)
diff --git a/docs/en/connectors/sink/Iceberg.md
b/docs/en/connectors/sink/Iceberg.md
index 509194f91b..f37246656c 100644
--- a/docs/en/connectors/sink/Iceberg.md
+++ b/docs/en/connectors/sink/Iceberg.md
@@ -89,6 +89,7 @@ Writing into an existing format version 3 table also accepts
`TIMESTAMP_NS` and
| custom_sql | string | Required when
`data_save_mode` is `CUSTOM_PROCESSING` | - | Custom
`delete` SQL for `CUSTOM_PROCESSING` data save mode, for example `delete from
... where ...`.
|
| iceberg.table.commit-branch | string | no | -
| Default branch for commits
|
| multi_table_sink_replica | int | no | 1
| Replica number for every table writer in multi-table sink mode.
Use this when one upstream job writes to multiple Iceberg tables and each table
needs more than one sink writer.
|
+| flush_row_size | long | no | -
| Rotate a sink writer after it has handled this many rows,
instead of keeping it open until the next checkpoint. Bounds the writer memory
of large upsert and CDC loads. Must be greater than 0. Please refer to
`flush_row_size` below.
|
| krb5_path | string | no | /etc/krb5.conf
| The path of `krb5.conf`, used for Kerberos authentication.
|
| kerberos_principal | string | no | -
| The principal for Kerberos authentication.
|
| kerberos_keytab_path | string | no | -
| The keytab file path for Kerberos authentication.
|
@@ -119,6 +120,27 @@ Use a comma-separated list, for example `dt,region` or
Iceberg transforms such a
Write commits to the specified Iceberg branch. Leave it empty to commit to the
table's default branch.
+### flush_row_size [long]
+
+By default a sink writer keeps its task writer open until the next checkpoint.
With a primary key (`iceberg.table.primary-keys`, upsert mode or a CDC source),
the task writer remembers every key it has written so that a later change of
the same row can be written as a position delete, and only frees them when it
is completed. A large batch load with a long checkpoint interval, or a batch
job that only checkpoints at the end, can therefore run out of memory.
+
+Set `flush_row_size` to complete the task writer after it has handled this
many rows and start a new one. This closes the current files early and releases
the remembered keys, so the writer memory no longer grows with the number of
rows per checkpoint. When it is not set, the writer is only rotated at
checkpoints, as before.
+
+Equality deletes written after a rotation must apply to rows written before
it, which Iceberg only does for files committed in an earlier snapshot. So when
a checkpoint contains equality deletes written after a rotation, it is
committed as several snapshots in the order the task writers were completed,
instead of one. Readers can see these snapshots one by one; each of them holds
a prefix of the changes of the checkpoint. Resuming a partly committed split
checkpoint is only supported on [...]
+
+Smaller values use less memory but produce more and smaller files. A rotation
completes the files of all open partitions of a partitioned table, not only of
the current one. In upsert mode every row writes an equality delete, so every
rotation becomes its own snapshot: a checkpoint is committed as about (rows per
writer in the checkpoint / `flush_row_size`) snapshots. CDC writes split a
checkpoint at each rotation that is followed by updates or deletes. Too small a
value can therefore bl [...]
+
+```hocon
+sink {
+ Iceberg {
+ ...
+ iceberg.table.primary-keys = "id"
+ iceberg.table.upsert-mode-enabled = true
+ flush_row_size = 1000000
+ }
+}
+```
+
### custom_sql [string]
When `data_save_mode = CUSTOM_PROCESSING`, configure the `delete` SQL that
removes the target data before the sink writes. This option is required in that
mode.
diff --git a/docs/en/introduction/concepts/incompatible-changes.md
b/docs/en/introduction/concepts/incompatible-changes.md
index 095e2d597a..ee520f7574 100644
--- a/docs/en/introduction/concepts/incompatible-changes.md
+++ b/docs/en/introduction/concepts/incompatible-changes.md
@@ -143,6 +143,14 @@ The same strictness applies wherever the shared
JSON-to-row conversion is used w
- **Impact**: The source type of these columns changes, for example from
`NVARCHAR(20)` to `NVARCHAR(10)`, and the column length used for sink
auto-create is halved (GB18030) or quartered (UTF-8), for example a MySQL sink
gets `VARCHAR(40)` instead of `VARCHAR(80)` for `NVARCHAR(10)`. Each new column
still holds every value of the source column, because the length is counted in
characters. Tables that already exist downstream are not altered. `CHAR`,
`VARCHAR`, `VARCHAR2` and `NCHAR` c [...]
- **Migration Guide**: No change is needed. Jobs that rely on the wider
auto-created columns, for example to write longer values from another source
into the same table, should create the sink table themselves.
+### Iceberg sink: a checkpoint with equality deletes after a writer rotation
is committed as several snapshots
+
+- **Behavior change: the Iceberg sink can commit one checkpoint as several
ordered snapshots**
+ - **Affected component**: `seatunnel-connectors-v2/connector-iceberg`
+ - **Description**: The sink committed all files of a checkpoint in one
snapshot. When the task writer was completed in the middle of a checkpoint,
which happens on a schema change with `iceberg.table.schema-evolution-enabled`
and now also with the new `flush_row_size` option, an equality delete written
after that point landed in the same snapshot as the row it targets. Iceberg
does not apply an equality delete to data files with the same sequence number,
so an upsert, update or delete [...]
+ - **Impact**: Only upsert and CDC writes that rotate the task writer inside
a checkpoint are affected; append-only writes and checkpoints without a
rotation still produce one snapshot. Readers can see the snapshots of such a
checkpoint one by one, and the snapshot count per checkpoint grows with the
number of rotations.
+ - **Migration Guide**: No configuration change is needed. Jobs restored from
a checkpoint committed by an older version are handled as before: a snapshot
with the checkpoint id and without `seatunnel.commit-group` is treated as the
whole checkpoint. If a job stops while a checkpoint that was split into several
snapshots is being committed, restore it on Zeta with this version or a later
one. An older version skips the whole checkpoint once it finds any of its
snapshots, and the Flink t [...]
+
### File sink: XML applies `date_format`, `datetime_format` and `time_format`
- **Behavior change: the XML writer of the file sinks formats DATE, TIME and
TIMESTAMP values with the configured formats**
diff --git a/docs/zh/connectors/sink/Iceberg.md
b/docs/zh/connectors/sink/Iceberg.md
index a42da85c98..a6596fddd1 100644
--- a/docs/zh/connectors/sink/Iceberg.md
+++ b/docs/zh/connectors/sink/Iceberg.md
@@ -89,6 +89,7 @@ libfb303-xxx.jar
| custom_sql | string | 当 `data_save_mode` 为
`CUSTOM_PROCESSING` 时是 | - | `CUSTOM_PROCESSING`
数据写入方式使用的自定义 `delete` SQL,例如 `delete from ... where ...`。
|
| iceberg.table.commit-branch | string | 否 | -
| 提交的默认分支
|
| multi_table_sink_replica | int | 否 | 1
| 多表写入模式下每张表对应的 Sink 写入并发数。一个作业写入多张 Iceberg 表,并且每张表都需要多个写入器时使用。
|
+| flush_row_size | long | 否 | -
| Sink 写入器处理这么多行后就轮换,不再一直保持到下一次 checkpoint,用来限制大批量 upsert 和 CDC
写入时的内存。必须大于 0。请参考下面的 `flush_row_size`。
|
| krb5_path | string | 否 | /etc/krb5.conf
| `krb5.conf` 文件的路径,用于 Kerberos 认证。
|
| kerberos_principal | string | 否 | -
| Kerberos 认证的 principal。
|
| kerberos_keytab_path | string | 否 | -
| Kerberos 认证的 keytab 文件路径。
|
@@ -119,6 +120,27 @@ iceberg.table.auto-create-props = {
将提交写入指定的 Iceberg 分支。不配置时写入表的默认分支。
+### flush_row_size [long]
+
+默认情况下,Sink 写入器会把当前的 task writer 一直保持到下一次 checkpoint。表有主键时(配置了
`iceberg.table.primary-keys`、开启 upsert 模式或上游是 CDC 源),task writer
会记住写过的每个主键,以便同一行后续的变更可以写成 position delete,这些主键要等 task writer 完成时才释放。因此
checkpoint 间隔很长的大批量导入,或者只在结束时做 checkpoint 的批作业,可能会内存溢出。
+
+设置 `flush_row_size` 后,task writer
每处理这么多行就完成一次并新建一个。这样会提前关闭当前文件并释放记住的主键,写入器内存不再随每个 checkpoint 的行数增长。不设置时只在
checkpoint 时轮换,与之前的行为一致。
+
+轮换之后写出的 equality delete 需要作用到轮换之前写入的行,而 Iceberg 只对更早快照中提交的文件应用 equality
delete。所以当一个 checkpoint 中包含轮换后写出的 equality delete 时,它会按 task writer
完成的顺序拆成多个快照依次提交,而不是一个。读端可能逐个看到这些快照,每个快照都包含该 checkpoint 变更的一个前缀。部分提交的拆分
checkpoint 只支持在 Zeta 上续提交:从该 checkpoint 恢复的作业只会补提交缺失的快照。Flink
翻译层不会调用这条恢复路径,未拆分的 checkpoint 也受影响。Flink 1.13 和 1.15 会丢弃恢复出来的全局
committable,其中缺失的快照不会被补提交,checkpoint 可能停留在部分已发布的状态。Flink 1.20
会把恢复出来的提交交给普通提交路径,该路径不会跳过表中已有的快照,可能再次提交它们的文件。可能拆分 checkpoint 的 upsert 和 CDC 作业(设
置了 `flush_row [...]
+
+取值越小内存占用越低,但文件更多更小。分区表轮换时会完成所有已打开分区的文件,而不只是当前分区。upsert 模式下每一行都会写 equality
delete,所以每次轮换都会单独成为一个快照:一个 checkpoint 大约提交(该 checkpoint 内单个写入器的行数 /
`flush_row_size`)个快照。CDC 写入在每次轮换后出现更新或删除时拆分
checkpoint。因此取值过小会让表元数据膨胀、查询规划变慢;请取较大的值,例如几十万到几百万行,并定期清理过期快照。
+
+```hocon
+sink {
+ Iceberg {
+ ...
+ iceberg.table.primary-keys = "id"
+ iceberg.table.upsert-mode-enabled = true
+ flush_row_size = 1000000
+ }
+}
+```
+
### custom_sql [string]
当 `data_save_mode = CUSTOM_PROCESSING` 时,配置在 Sink 写入前删除目标数据的 `delete`
SQL。该模式下必须配置此选项。
diff --git a/docs/zh/introduction/concepts/incompatible-changes.md
b/docs/zh/introduction/concepts/incompatible-changes.md
index 1ca3c44374..12bef03fc3 100644
--- a/docs/zh/introduction/concepts/incompatible-changes.md
+++ b/docs/zh/introduction/concepts/incompatible-changes.md
@@ -132,6 +132,14 @@ MySQL JDBC Sink 现在直接绑定 SeaTunnel
`TIMESTAMP`(`LocalDateTime`),
- **影响**:这类列的源类型会变化,例如从 `NVARCHAR(20)` 变为 `NVARCHAR(10)`;Sink
自动建表使用的列长度变为原来的 1/2(GB18030)或 1/4(UTF-8),例如 `NVARCHAR(10)` 写入 MySQL Sink 时建为
`VARCHAR(40)`,而不是
`VARCHAR(80)`。由于长度按字符计算,新列仍能容纳源列的所有值。下游已存在的表不会被修改。`CHAR`、`VARCHAR`、`VARCHAR2` 和
`NCHAR` 列不受影响。
- **迁移指南**:无需修改。如果作业依赖原来较宽的自动建表列(例如还要从其他数据源向同一张表写入更长的值),请自行创建 Sink 表。
+### Iceberg Sink:writer 轮换后带有 equality delete 的 checkpoint 拆成多个快照提交
+
+- **行为变更:Iceberg Sink 可能把一个 checkpoint 按顺序提交为多个快照**
+ - **影响组件**:`seatunnel-connectors-v2/connector-iceberg`
+ - **描述**:此前 Sink 把一个 checkpoint 的所有文件放在一个快照中提交。当 task writer 在 checkpoint
中途完成时(开启 `iceberg.table.schema-evolution-enabled` 后发生 schema 变更,以及现在新增的
`flush_row_size` 选项),之后写出的 equality delete 会和它要删除的行落在同一个快照里。Iceberg 不会把
equality delete 应用到相同序列号的数据文件上,因此对同一 checkpoint 中先前写入的行做
upsert、更新或删除时,旧行会留在表中。现在这样的 checkpoint 会按 task writer 完成的顺序拆成多个快照提交。每个快照带有
`seatunnel.checkpoint-id`,以及 `seatunnel.commit-group` 和
`seatunnel.commit-group-count`。
+ - **影响**:只影响在 checkpoint 内轮换了 task writer 的 upsert 和 CDC 写入;纯追加写入和没有轮换的
checkpoint 仍然只产生一个快照。读端可能逐个看到这样一个 checkpoint 的多个快照,每个 checkpoint 的快照数随轮换次数增加。
+ - **迁移指南**:无需修改配置。从旧版本提交的 checkpoint 恢复的作业处理方式不变:带有 checkpoint id 但没有
`seatunnel.commit-group` 的快照视为整个 checkpoint。如果作业在提交一个被拆成多个快照的 checkpoint 时中断,请在
Zeta 上用当前或更新的版本恢复。旧版本只要找到该 checkpoint 的任意一个快照就会跳过整个 checkpoint,Flink 翻译层也不会调用
Sink 的恢复路径,这两种情况下剩下的快照都无法正确续提交;详见 Iceberg Sink 文档中的 `flush_row_size`。
+
### File Sink:XML 应用 `date_format`、`datetime_format` 和 `time_format`
- **行为变更:文件类 Sink 的 XML 写入器按配置的格式输出 DATE、TIME 和 TIMESTAMP**
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfig.java
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfig.java
index 6567208e4e..5c5a0b247d 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfig.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfig.java
@@ -21,6 +21,7 @@ import
org.apache.seatunnel.shade.com.google.common.annotations.VisibleForTestin
import org.apache.seatunnel.shade.com.google.common.collect.ImmutableList;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.configuration.util.OptionValidationException;
import org.apache.seatunnel.api.sink.DataSaveMode;
import org.apache.seatunnel.api.sink.SchemaSaveMode;
@@ -61,6 +62,8 @@ public class IcebergSinkConfig extends IcebergCommonConfig {
private SchemaSaveMode schemaSaveMode;
private DataSaveMode dataSaveMode;
private String dataSaveModeSQL;
+ /** Rotate the task writer after this many rows; null keeps it open until
the checkpoint. */
+ private Long flushRowSize;
public IcebergSinkConfig(ReadonlyConfig readonlyConfig) {
super(readonlyConfig);
@@ -81,6 +84,12 @@ public class IcebergSinkConfig extends IcebergCommonConfig {
this.dataSaveMode =
readonlyConfig.get(IcebergSinkOptions.DATA_SAVE_MODE);
this.dataSaveModeSQL =
readonlyConfig.get(IcebergSinkOptions.DATA_SAVE_MODE_CUSTOM_SQL);
this.commitBranch =
readonlyConfig.get(IcebergSinkOptions.TABLES_DEFAULT_COMMIT_BRANCH);
+ this.flushRowSize =
readonlyConfig.get(IcebergSinkOptions.FLUSH_ROW_SIZE);
+ if (flushRowSize != null && flushRowSize <= 0) {
+ throw new OptionValidationException(
+ "The option '%s' must be greater than 0, but was %d.",
+ IcebergSinkOptions.FLUSH_ROW_SIZE.key(), flushRowSize);
+ }
}
@VisibleForTesting
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkOptions.java
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkOptions.java
index 38204b042b..5f0c263bea 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkOptions.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkOptions.java
@@ -96,4 +96,14 @@ public class IcebergSinkOptions extends IcebergCommonOptions
{
.stringType()
.noDefaultValue()
.withDescription("Default branch for commits");
+
+ public static final Option<Long> FLUSH_ROW_SIZE =
+ Options.key("flush_row_size")
+ .longType()
+ .noDefaultValue()
+ .withDescription(
+ "Maximum number of rows a sink writer handles
before it closes its"
+ + " current files and starts new ones,
instead of keeping them"
+ + " open until the next checkpoint. Must
be greater than 0."
+ + " Not set means the writer is only
rotated at checkpoints.");
}
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/IcebergSinkFactory.java
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/IcebergSinkFactory.java
index 1eb1e4c8dd..5466248838 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/IcebergSinkFactory.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/IcebergSinkFactory.java
@@ -67,7 +67,8 @@ public class IcebergSinkFactory implements TableSinkFactory {
IcebergSinkOptions.TABLE_UPSERT_MODE_ENABLED_PROP,
IcebergSinkOptions.TABLE_SCHEMA_EVOLUTION_ENABLED_PROP,
IcebergSinkOptions.TABLES_DEFAULT_COMMIT_BRANCH,
- SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA)
+ SinkConnectorCommonOptions.MULTI_TABLE_SINK_REPLICA,
+ IcebergSinkOptions.FLUSH_ROW_SIZE)
.conditional(
IcebergSinkOptions.DATA_SAVE_MODE,
DataSaveMode.CUSTOM_PROCESSING,
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergAggregatedCommitter.java
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergAggregatedCommitter.java
index 4ec84be313..1e148b233c 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergAggregatedCommitter.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergAggregatedCommitter.java
@@ -23,12 +23,16 @@ import
org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergTableLoader;
import
org.apache.seatunnel.connectors.seatunnel.iceberg.config.IcebergSinkConfig;
import
org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.WriteResult;
+import org.apache.iceberg.FileContent;
+
import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
/** Iceberg aggregated committer */
@Slf4j
@@ -55,24 +59,25 @@ public class IcebergAggregatedCommitter
public List<IcebergAggregatedCommitInfo> commit(
List<IcebergAggregatedCommitInfo> aggregatedCommitInfo) throws
IOException {
for (IcebergAggregatedCommitInfo commitInfo : aggregatedCommitInfo) {
- commitFiles(commitInfo.getCommitInfos(),
commitInfo.getCheckpointId());
+ commitFiles(commitInfo.getCommitInfos(),
commitInfo.getCheckpointId(), false);
}
return Collections.emptyList();
}
+ /**
+ * Commits restored checkpoints, skipping the snapshots that already
reached the table. For a
+ * checkpoint that was split into several snapshots, only the missing
groups are committed.
+ *
+ * <p>Resuming a partially committed split checkpoint requires the engine
to call this method
+ * with the recovered committables. Zeta does. The Flink 1.13 and 1.15
adapters discard
+ * recovered global committables, and the Flink 1.20 adapter passes
recovered pending
+ * committables to {@link #commit(List)} instead. See the {@code
flush_row_size} documentation.
+ */
@Override
public List<IcebergAggregatedCommitInfo> restoreCommit(
List<IcebergAggregatedCommitInfo> aggregatedCommitInfo) throws
IOException {
for (IcebergAggregatedCommitInfo commitInfo : aggregatedCommitInfo) {
- long checkpointId = commitInfo.getCheckpointId();
- List<WriteResult> allResults =
flattenResults(commitInfo.getCommitInfos());
- if (filesCommitter.isAlreadyCommitted(checkpointId, allResults)) {
- log.info(
- "Checkpoint {} already committed to Iceberg table,
skipping restore commit",
- checkpointId);
- continue;
- }
- commitFiles(commitInfo.getCommitInfos(), checkpointId);
+ commitFiles(commitInfo.getCommitInfos(),
commitInfo.getCheckpointId(), true);
}
return Collections.emptyList();
}
@@ -90,17 +95,90 @@ public class IcebergAggregatedCommitter
this.tableLoader.close();
}
- private void commitFiles(List<IcebergCommitInfo> commitInfos, long
checkpointId) {
- List<WriteResult> allResults = flattenResults(commitInfos);
- if (!allResults.isEmpty()) {
- filesCommitter.doCommit(allResults, checkpointId);
+ private void commitFiles(
+ List<IcebergCommitInfo> commitInfos, long checkpointId, boolean
restoring) {
+ List<List<WriteResult>> groups = planCommits(commitInfos);
+ if (groups.isEmpty()) {
+ return;
+ }
+ if (groups.size() > 1) {
+ log.info(
+ "Checkpoint {} has equality deletes after a writer
rotation, committing it as"
+ + " {} ordered snapshots",
+ checkpointId,
+ groups.size());
+ filesCommitter.doCommitInOrder(groups, checkpointId, restoring);
+ return;
}
+ List<WriteResult> results = groups.get(0);
+ if (restoring && filesCommitter.isAlreadyCommitted(checkpointId,
results)) {
+ log.info(
+ "Checkpoint {} already committed to Iceberg table,
skipping restore commit",
+ checkpointId);
+ return;
+ }
+ filesCommitter.doCommit(results, checkpointId);
+ }
+
+ /**
+ * Splits the results of one checkpoint into groups that must be committed
one after another.
+ *
+ * <p>Each writer reports one result per task writer it completed, in
order. Iceberg applies an
+ * equality delete only to data files with a smaller sequence number, so
an equality delete
+ * written after a rotation does not remove a row written before it if
both land in the same
+ * snapshot. The i-th results of all writers form round i, and a round
that carries equality
+ * deletes starts a new group. Without rotation, or without equality
deletes, there is a single
+ * group, which is the same single snapshot as before.
+ */
+ static List<List<WriteResult>> planCommits(List<IcebergCommitInfo>
commitInfos) {
+ List<List<WriteResult>> perWriter =
+ commitInfos.stream()
+ .map(IcebergCommitInfo::getResults)
+ .filter(results -> results != null &&
!results.isEmpty())
+ .collect(Collectors.toList());
+ int rounds = perWriter.stream().mapToInt(List::size).max().orElse(0);
+ List<List<WriteResult>> groups = new ArrayList<>();
+ for (int round = 0; round < rounds; round++) {
+ List<WriteResult> roundResults = new ArrayList<>();
+ for (List<WriteResult> results : perWriter) {
+ if (round < results.size()) {
+ roundResults.add(results.get(round));
+ }
+ }
+ // A group without records (for example from a flush right after a
schema change)
+ // commits no snapshot, so later rounds join it instead of opening
another group.
+ if (groups.isEmpty()
+ || (hasEqualityDeletes(roundResults)
+ && hasRecords(groups.get(groups.size() - 1)))) {
+ groups.add(roundResults);
+ } else {
+ groups.get(groups.size() - 1).addAll(roundResults);
+ }
+ }
+ return groups;
+ }
+
+ private static boolean hasRecords(List<WriteResult> results) {
+ return results.stream()
+ .flatMap(
+ result ->
+ Stream.concat(
+ result.getDataFiles() == null
+ ? Stream.empty()
+ :
result.getDataFiles().stream(),
+ result.getDeleteFiles() == null
+ ? Stream.empty()
+ :
result.getDeleteFiles().stream()))
+ .anyMatch(file -> file.recordCount() > 0);
}
- private List<WriteResult> flattenResults(List<IcebergCommitInfo>
commitInfos) {
- return commitInfos.stream()
- .filter(info -> info.getResults() != null &&
!info.getResults().isEmpty())
- .flatMap(info -> info.getResults().stream())
- .collect(Collectors.toList());
+ private static boolean hasEqualityDeletes(List<WriteResult> results) {
+ return results.stream()
+ .filter(result -> result.getDeleteFiles() != null)
+ .flatMap(result -> result.getDeleteFiles().stream())
+ .anyMatch(
+ file ->
+ file.content() == FileContent.EQUALITY_DELETES
+ && file.recordCount() > 0);
}
}
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergFilesCommitter.java
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergFilesCommitter.java
index 849f592328..69a98abbbb 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergFilesCommitter.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergFilesCommitter.java
@@ -38,8 +38,10 @@ import lombok.extern.slf4j.Slf4j;
import java.io.IOException;
import java.io.Serializable;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import static java.util.stream.Collectors.toList;
@@ -51,6 +53,16 @@ public class IcebergFilesCommitter implements Serializable {
public static final String SNAPSHOT_PROPERTY_CHECKPOINT_ID =
"seatunnel.checkpoint-id";
+ /**
+ * Set when one checkpoint is committed as several ordered snapshots: the
0-based position of
+ * this snapshot, and how many there are. A snapshot that carries the
checkpoint id but no group
+ * index covers the whole checkpoint.
+ */
+ public static final String SNAPSHOT_PROPERTY_COMMIT_GROUP =
"seatunnel.commit-group";
+
+ public static final String SNAPSHOT_PROPERTY_COMMIT_GROUP_COUNT =
+ "seatunnel.commit-group-count";
+
private IcebergTableLoader icebergTableLoader;
private boolean caseSensitive;
private String branch;
@@ -68,7 +80,87 @@ public class IcebergFilesCommitter implements Serializable {
public void doCommit(List<WriteResult> results, long checkpointId) {
TableIdentifier tableIdentifier =
icebergTableLoader.getTableIdentifier();
- commit(tableIdentifier, results, checkpointId);
+ commit(tableIdentifier, results, checkpointId, -1, 1);
+ }
+
+ /**
+ * Commits the groups of one checkpoint as separate snapshots, in order,
so that equality
+ * deletes of a later group get a higher sequence number than the data
files of the earlier
+ * ones. When restoring, groups that already reached the table are skipped.
+ */
+ public void doCommitInOrder(
+ List<List<WriteResult>> groups, long checkpointId, boolean
skipCommittedGroups) {
+ TableIdentifier tableIdentifier =
icebergTableLoader.getTableIdentifier();
+ Set<Integer> committed =
+ skipCommittedGroups
+ ? committedGroups(checkpointId, groups)
+ : Collections.emptySet();
+ for (int group = 0; group < groups.size(); group++) {
+ if (committed.contains(group)) {
+ log.info(
+ "Group {}/{} of checkpoint {} already committed to
Iceberg table, skipping",
+ group + 1,
+ groups.size(),
+ checkpointId);
+ continue;
+ }
+ commit(tableIdentifier, groups.get(group), checkpointId, group,
groups.size());
+ }
+ }
+
+ /**
+ * Returns the indexes of the groups of a checkpoint that already reached
the table, so that a
+ * restore commits only the missing ones.
+ *
+ * <ul>
+ * <li>A non-positive checkpoint id cannot be matched against snapshot
summaries, so each
+ * group is checked by looking for its file paths in the head
snapshot.
+ * <li>A positive checkpoint id is matched against {@link
#SNAPSHOT_PROPERTY_CHECKPOINT_ID}
+ * while walking back from the head of the selected branch (or
main), and the group index
+ * is read from {@link #SNAPSHOT_PROPERTY_COMMIT_GROUP}. The walk
stops early once every
+ * group has been found.
+ * <li>A snapshot with this checkpoint id but no group index was written
as a single,
+ * ungrouped snapshot (for example before this checkpoint was split
into groups), so the
+ * whole checkpoint is treated as committed.
+ * </ul>
+ *
+ * <p>Group indexes are only meaningful because {@code planCommits} builds
the same groups, in
+ * the same order, from the results persisted in the checkpoint state.
+ */
+ private Set<Integer> committedGroups(long checkpointId,
List<List<WriteResult>> groups) {
+ Set<Integer> committed = new HashSet<>();
+ if (checkpointId <= 0) {
+ for (int group = 0; group < groups.size(); group++) {
+ if (areFilesAlreadyCommitted(groups.get(group))) {
+ committed.add(group);
+ }
+ }
+ return committed;
+ }
+ Table table = icebergTableLoader.loadTable();
+ String expected = String.valueOf(checkpointId);
+ Snapshot current = headSnapshot(table);
+ while (current != null) {
+ Map<String, String> summary = current.summary();
+ if (expected.equals(summary.get(SNAPSHOT_PROPERTY_CHECKPOINT_ID)))
{
+ String group = summary.get(SNAPSHOT_PROPERTY_COMMIT_GROUP);
+ if (group == null) {
+ // committed as a single snapshot before the job was
restarted
+ for (int i = 0; i < groups.size(); i++) {
+ committed.add(i);
+ }
+ return committed;
+ }
+ committed.add(Integer.parseInt(group));
+ if (committed.size() == groups.size()) {
+ // every group is already in the table, no need to walk
older history
+ return committed;
+ }
+ }
+ Long parentId = current.parentId();
+ current = parentId != null ? table.snapshot(parentId) : null;
+ }
+ return committed;
}
public boolean isAlreadyCommitted(long checkpointId, List<WriteResult>
results) {
@@ -157,7 +249,11 @@ public class IcebergFilesCommitter implements Serializable
{
}
private void commit(
- TableIdentifier tableIdentifier, List<WriteResult> results, long
checkpointId) {
+ TableIdentifier tableIdentifier,
+ List<WriteResult> results,
+ long checkpointId,
+ int group,
+ int groupCount) {
List<DataFile> dataFiles =
results.stream()
.filter(payload -> payload.getDataFiles() != null)
@@ -185,6 +281,10 @@ public class IcebergFilesCommitter implements Serializable
{
if (checkpointId > 0) {
append.set(SNAPSHOT_PROPERTY_CHECKPOINT_ID,
String.valueOf(checkpointId));
}
+ if (group >= 0) {
+ append.set(SNAPSHOT_PROPERTY_COMMIT_GROUP,
String.valueOf(group));
+ append.set(SNAPSHOT_PROPERTY_COMMIT_GROUP_COUNT,
String.valueOf(groupCount));
+ }
dataFiles.forEach(append::appendFile);
append.commit();
} else {
@@ -196,6 +296,10 @@ public class IcebergFilesCommitter implements Serializable
{
if (checkpointId > 0) {
delta.set(SNAPSHOT_PROPERTY_CHECKPOINT_ID,
String.valueOf(checkpointId));
}
+ if (group >= 0) {
+ delta.set(SNAPSHOT_PROPERTY_COMMIT_GROUP,
String.valueOf(group));
+ delta.set(SNAPSHOT_PROPERTY_COMMIT_GROUP_COUNT,
String.valueOf(groupCount));
+ }
dataFiles.forEach(delta::addRows);
deleteFiles.forEach(delta::addDeletes);
delta.commit();
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/IcebergRecordWriter.java
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/IcebergRecordWriter.java
index 7c30487d52..caac0b2a27 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/IcebergRecordWriter.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/IcebergRecordWriter.java
@@ -58,6 +58,8 @@ public class IcebergRecordWriter implements RecordWriter {
private volatile TaskWriter<Record> writer;
private RowConverter recordConverter;
private final IcebergWriterFactory writerFactory;
+ /** Rows written by the current task writer; it is rotated when this
reaches the limit. */
+ private long rowsInWriter;
public IcebergRecordWriter(
Table table, IcebergWriterFactory writerFactory, IcebergSinkConfig
config) {
@@ -91,6 +93,15 @@ public class IcebergRecordWriter implements RecordWriter {
} catch (IOException e) {
throw new RuntimeException(e);
}
+ rowsInWriter++;
+ Long flushRowSize = config.getFlushRowSize();
+ if (flushRowSize != null && rowsInWriter >= flushRowSize) {
+ // A delta writer remembers every key it has inserted until it
completes, so a long
+ // checkpoint interval keeps all of them in memory. Completing it
here frees that
+ // memory. Each completed writer becomes its own WriteResult, and
the committer keeps
+ // their order so that later equality deletes still apply to
earlier rows.
+ flush();
+ }
}
@Override
@@ -191,5 +202,6 @@ public class IcebergRecordWriter implements RecordWriter {
Arrays.asList(writeResult.deleteFiles()),
table.spec().partitionType()));
writer = null;
+ rowsInWriter = 0;
}
}
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/RecordWriter.java
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/RecordWriter.java
index fb37cccc1f..189c28f8cf 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/RecordWriter.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/main/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/writer/RecordWriter.java
@@ -32,6 +32,11 @@ public interface RecordWriter extends Cloneable {
void applySchemaChange(SeaTunnelRowType afterRowType, SchemaChangeEvent
event);
+ /**
+ * Completes the open task writer and returns the results of every task
writer completed since
+ * the previous call, in the order they were completed. The committer
relies on this order:
+ * equality deletes in a later result may target rows of an earlier one.
+ */
default List<WriteResult> complete() {
return ImmutableList.of();
}
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfigTest.java
b/seatunnel-connectors-v2/connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfigTest.java
index 2353fbffda..3d859182ea 100644
---
a/seatunnel-connectors-v2/connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfigTest.java
+++
b/seatunnel-connectors-v2/connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/config/IcebergSinkConfigTest.java
@@ -18,9 +18,12 @@
package org.apache.seatunnel.connectors.seatunnel.iceberg.config;
import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.configuration.util.OptionValidationException;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
import java.util.Arrays;
import java.util.HashMap;
@@ -42,4 +45,34 @@ public class IcebergSinkConfigTest {
Arrays.asList("bucket(id, 16)", "truncate(col, 8)", "dt"),
config.getPartitionKeys());
}
+
+ @Test
+ public void testFlushRowSizeIsUnsetByDefault() {
+ Assertions.assertNull(new
IcebergSinkConfig(flushRowSizeConfig(null)).getFlushRowSize());
+ Assertions.assertEquals(
+ 1000L,
+ new
IcebergSinkConfig(flushRowSizeConfig(1000L)).getFlushRowSize().longValue());
+ }
+
+ @ParameterizedTest
+ @ValueSource(longs = {0L, -1L})
+ public void testFlushRowSizeMustBePositive(long flushRowSize) {
+ OptionValidationException exception =
+ Assertions.assertThrows(
+ OptionValidationException.class,
+ () -> new
IcebergSinkConfig(flushRowSizeConfig(flushRowSize)));
+ Assertions.assertTrue(
+ exception.getMessage().contains("'flush_row_size' must be
greater than 0"),
+ exception.getMessage());
+ }
+
+ private static ReadonlyConfig flushRowSizeConfig(Long flushRowSize) {
+ Map<String, Object> configs = new HashMap<>();
+ configs.put(IcebergCommonOptions.KEY_TABLE.key(), "tbl");
+ configs.put(IcebergCommonOptions.CATALOG_PROPS.key(), new
HashMap<String, String>());
+ if (flushRowSize != null) {
+ configs.put(IcebergSinkOptions.FLUSH_ROW_SIZE.key(), flushRowSize);
+ }
+ return ReadonlyConfig.fromMap(configs);
+ }
}
diff --git
a/seatunnel-connectors-v2/connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergWriterRotationTest.java
b/seatunnel-connectors-v2/connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergWriterRotationTest.java
new file mode 100644
index 0000000000..6efe3a3a87
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-iceberg/src/test/java/org/apache/seatunnel/connectors/seatunnel/iceberg/sink/commit/IcebergWriterRotationTest.java
@@ -0,0 +1,437 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.iceberg.sink.commit;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
+import org.apache.seatunnel.api.table.schema.event.AlterTableAddColumnEvent;
+import org.apache.seatunnel.api.table.type.BasicType;
+import org.apache.seatunnel.api.table.type.RowKind;
+import org.apache.seatunnel.api.table.type.SeaTunnelDataType;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
+import org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergTableLoader;
+import
org.apache.seatunnel.connectors.seatunnel.iceberg.config.IcebergCommonOptions;
+import
org.apache.seatunnel.connectors.seatunnel.iceberg.config.IcebergSinkConfig;
+import
org.apache.seatunnel.connectors.seatunnel.iceberg.config.IcebergSinkOptions;
+import
org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.IcebergWriterFactory;
+import
org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.RecordWriter;
+import
org.apache.seatunnel.connectors.seatunnel.iceberg.sink.writer.WriteResult;
+import org.apache.seatunnel.connectors.seatunnel.iceberg.utils.SchemaUtils;
+
+import org.apache.iceberg.RowDelta;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.data.IcebergGenerics;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.io.CloseableIterable;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Writes through a real task writer with {@code flush_row_size} set and
commits through the
+ * aggregated committer, then reads the table back with Iceberg's own reader.
+ */
+class IcebergWriterRotationTest {
+
+ private static final TableIdentifier TABLE = TableIdentifier.of("db",
"events");
+
+ private InMemoryCatalog catalog;
+
+ @BeforeEach
+ void setUp() {
+ catalog = new InMemoryCatalog();
+ catalog.initialize("test", Collections.emptyMap());
+ catalog.createNamespace(Namespace.of("db"));
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ catalog.close();
+ }
+
+ @Test
+ void appendOnlyRotationKeepsOneSnapshotPerCheckpoint() throws Exception {
+ IcebergSinkConfig config = config(2L, false, null);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+
+ List<WriteResult> results =
+ write(table, config, insert(1, "a"), insert(2, "b"), insert(3,
"c"));
+
+ assertEquals(2, results.size(), "two rows per task writer, so three
rows need two");
+ commit(table, config, 1L, results);
+ assertEquals(rows("1=a", "2=b", "3=c"), read(table));
+ assertEquals(1, snapshotCount(table));
+ assertNull(
+ table.currentSnapshot()
+ .summary()
+
.get(IcebergFilesCommitter.SNAPSHOT_PROPERTY_COMMIT_GROUP));
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"", "bucket(id, 4)"})
+ void upsertDeleteAndUpdateAfterRotationApplyToEarlierRows(String
partitionKeys)
+ throws Exception {
+ IcebergSinkConfig config = config(2L, true, partitionKeys);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+
+ // the writer is rotated after the second row, so key 1 and key 2 are
inserted by the
+ // first task writer and deleted or updated by the second one
+ List<WriteResult> results =
+ write(
+ table,
+ config,
+ insert(1, "a"),
+ insert(2, "b"),
+ row(RowKind.DELETE, 1, "a"),
+ row(RowKind.UPDATE_AFTER, 2, "b2"),
+ insert(3, "c"));
+
+ assertEquals(3, results.size());
+ commit(table, config, 1L, results);
+
+ assertEquals(rows("2=b2", "3=c"), read(table));
+ assertEquals(3, snapshotCount(table));
+ assertEquals(
+ "3",
+ table.currentSnapshot()
+ .summary()
+
.get(IcebergFilesCommitter.SNAPSHOT_PROPERTY_COMMIT_GROUP_COUNT));
+ }
+
+ @Test
+ void resultWithoutRecordsDoesNotOpenAGroup() throws Exception {
+ IcebergSinkConfig config = config(2L, true, null);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+ List<WriteResult> written =
+ write(
+ table,
+ config,
+ insert(1, "a"),
+ insert(2, "b"),
+ row(RowKind.DELETE, 1, "a"),
+ row(RowKind.UPDATE_AFTER, 2, "b2"),
+ insert(3, "c"));
+
+ // a task writer completed without rows, as after a schema change with
nothing written
+ List<WriteResult> results = new ArrayList<>();
+ results.add(
+ new WriteResult(
+ Collections.emptyList(),
+ Collections.emptyList(),
+ table.spec().partitionType()));
+ results.addAll(written);
+ commit(table, config, 1L, results);
+
+ assertEquals(rows("2=b2", "3=c"), read(table));
+ List<String> groups = new ArrayList<>();
+ for (Snapshot snapshot : table.snapshots()) {
+ groups.add(
+
snapshot.summary().get(IcebergFilesCommitter.SNAPSHOT_PROPERTY_COMMIT_GROUP)
+ + "/"
+ + snapshot.summary()
+ .get(
+ IcebergFilesCommitter
+
.SNAPSHOT_PROPERTY_COMMIT_GROUP_COUNT));
+ }
+ assertEquals(Arrays.asList("0/3", "1/3", "2/3"), groups);
+ }
+
+ @Test
+ void singleSnapshotForRotatedUpsertResultsLeavesStaleRow() throws
Exception {
+ // Shows why the committer keeps the order: committing the same files
as one snapshot,
+ // as the committer did before, keeps the deleted and the replaced row.
+ IcebergSinkConfig config = config(2L, true, null);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+
+ List<WriteResult> results =
+ write(
+ table,
+ config,
+ insert(1, "a"),
+ insert(2, "b"),
+ row(RowKind.DELETE, 1, "a"),
+ row(RowKind.UPDATE_AFTER, 2, "b2"));
+ RowDelta rowDelta = table.newRowDelta();
+ for (WriteResult result : results) {
+ result.getDataFiles().forEach(rowDelta::addRows);
+ result.getDeleteFiles().forEach(rowDelta::addDeletes);
+ }
+ rowDelta.commit();
+
+ assertEquals(rows("1=a", "2=b", "2=b2"), read(table));
+ }
+
+ @Test
+ void cdcUpdateAfterRotationReplacesEarlierRow() throws Exception {
+ Map<String, Object> options = new HashMap<>();
+ options.put(IcebergSinkOptions.TABLE_PRIMARY_KEYS.key(), "id");
+ options.put(IcebergSinkOptions.FLUSH_ROW_SIZE.key(), 1L);
+ IcebergSinkConfig config = sinkConfig(options);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+
+ List<WriteResult> results =
+ write(
+ table,
+ config,
+ insert(1, "a"),
+ row(RowKind.UPDATE_BEFORE, 1, "a"),
+ row(RowKind.UPDATE_AFTER, 1, "a2"));
+
+ commit(table, config, 1L, results);
+ assertEquals(rows("1=a2"), read(table));
+ }
+
+ @Test
+ void parallelWritersWithDifferentRotationCounts() throws Exception {
+ IcebergSinkConfig config = config(1L, true, null);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+
+ List<WriteResult> writer0 =
+ write(
+ table,
+ config,
+ insert(1, "a"),
+ row(RowKind.UPDATE_AFTER, 1, "a2"),
+ row(RowKind.DELETE, 1, "a2"));
+ List<WriteResult> writer1 = write(table, config, insert(2, "b"));
+
+ IcebergAggregatedCommitter committer = committer(table, config);
+ committer.commit(
+ Collections.singletonList(
+ committer.combine(
+ Arrays.asList(
+ new IcebergCommitInfo(writer0, 1L),
+ new IcebergCommitInfo(writer1, 1L)))));
+
+ assertEquals(rows("2=b"), read(table));
+ }
+
+ @Test
+ void restoreCommitsOnlyGroupsMissingFromTable() throws Exception {
+ IcebergSinkConfig config = config(2L, true, null);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+ List<WriteResult> results =
+ write(
+ table,
+ config,
+ insert(1, "a"),
+ insert(2, "b"),
+ row(RowKind.DELETE, 1, "a"),
+ insert(3, "c"));
+ List<IcebergCommitInfo> infos =
+ Collections.singletonList(new IcebergCommitInfo(results, 7L));
+ List<List<WriteResult>> groups =
IcebergAggregatedCommitter.planCommits(infos);
+ assertEquals(2, groups.size());
+
+ // the job failed after the first snapshot of checkpoint 7 reached the
table
+ IcebergFilesCommitter.of(config, loader(table))
+ .doCommitInOrder(groups.subList(0, 1), 7L, false);
+ assertEquals(rows("1=a", "2=b"), read(table));
+
+ IcebergAggregatedCommitter committer = committer(table, config);
+ IcebergAggregatedCommitInfo aggregated = committer.combine(infos);
+ committer.restoreCommit(Collections.singletonList(aggregated));
+ assertEquals(rows("2=b", "3=c"), read(table));
+ assertEquals(2, snapshotCount(table));
+
+ // restoring the same checkpoint again changes nothing
+ committer.restoreCommit(Collections.singletonList(aggregated));
+ assertEquals(2, snapshotCount(table));
+ assertEquals(rows("2=b", "3=c"), read(table));
+ }
+
+ @Test
+ void restoreSkipsCheckpointCommittedAsSingleSnapshot() throws Exception {
+ IcebergSinkConfig config = config(2L, true, null);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+ List<WriteResult> results =
+ write(table, config, insert(1, "a"), insert(2, "b"),
row(RowKind.DELETE, 1, "a"));
+ // a snapshot written for checkpoint 7 without group properties covers
all of it
+ IcebergFilesCommitter.of(config, loader(table)).doCommit(results, 7L);
+ long snapshots = snapshotCount(table);
+
+ IcebergAggregatedCommitter committer = committer(table, config);
+ committer.restoreCommit(
+ Collections.singletonList(
+ committer.combine(
+ Collections.singletonList(new
IcebergCommitInfo(results, 7L)))));
+
+ assertEquals(snapshots, snapshotCount(table));
+ }
+
+ @Test
+ void schemaChangeInsideCheckpointKeepsUpsertCorrect() throws Exception {
+ // Applying a schema change completes the task writer as well, so this
was affected by the
+ // same problem before rotation existed.
+ IcebergSinkConfig config = config(null, true, null);
+ Table table = SchemaUtils.autoCreateTable(catalog, TABLE, config,
idAndName());
+ RecordWriter writer = new IcebergWriterFactory(loader(table),
config).createWriter(null);
+ SeaTunnelRowType before = idAndName().toPhysicalRowDataType();
+ writer.write(insert(1, "a"), before);
+
+ SeaTunnelRowType after =
+ new SeaTunnelRowType(
+ new String[] {"id", "name", "note"},
+ new SeaTunnelDataType[] {
+ BasicType.LONG_TYPE, BasicType.STRING_TYPE,
BasicType.STRING_TYPE
+ });
+ writer.applySchemaChange(
+ after,
+ AlterTableAddColumnEvent.add(
+
org.apache.seatunnel.api.table.catalog.TableIdentifier.of(
+ "test", "db", "events"),
+ PhysicalColumn.of(
+ "note", BasicType.STRING_TYPE, (Long) null,
true, null, null)));
+ SeaTunnelRow update = new SeaTunnelRow(new Object[] {1L, "a2", "n"});
+ update.setRowKind(RowKind.UPDATE_AFTER);
+ writer.write(update, after);
+
+ commit(table, config, 1L, writer.complete());
+ assertEquals(rows("1=a2"), read(table));
+ }
+
+ private void commit(
+ Table table, IcebergSinkConfig config, long checkpointId,
List<WriteResult> results)
+ throws Exception {
+ IcebergAggregatedCommitter committer = committer(table, config);
+ committer.commit(
+ Collections.singletonList(
+ committer.combine(
+ Collections.singletonList(
+ new IcebergCommitInfo(results,
checkpointId)))));
+ }
+
+ private static IcebergAggregatedCommitter committer(Table table,
IcebergSinkConfig config)
+ throws Exception {
+ IcebergAggregatedCommitter committer = new
IcebergAggregatedCommitter(config, null);
+ Field field =
IcebergAggregatedCommitter.class.getDeclaredField("filesCommitter");
+ field.setAccessible(true);
+ field.set(committer, IcebergFilesCommitter.of(config, loader(table)));
+ return committer;
+ }
+
+ private static List<WriteResult> write(
+ Table table, IcebergSinkConfig config, SeaTunnelRow... rows) {
+ RecordWriter writer = new IcebergWriterFactory(loader(table),
config).createWriter(null);
+ SeaTunnelRowType rowType = idAndName().toPhysicalRowDataType();
+ for (SeaTunnelRow row : rows) {
+ writer.write(row, rowType);
+ }
+ return writer.complete();
+ }
+
+ /** Rows as "id=name", sorted, keeping duplicates. */
+ private static List<String> read(Table table) throws IOException {
+ table.refresh();
+ List<String> rows = new ArrayList<>();
+ try (CloseableIterable<Record> records =
IcebergGenerics.read(table).build()) {
+ for (Record record : records) {
+ rows.add(record.getField("id") + "=" +
record.getField("name"));
+ }
+ }
+ Collections.sort(rows);
+ return rows;
+ }
+
+ private static List<String> rows(String... rows) {
+ List<String> sorted = new ArrayList<>(Arrays.asList(rows));
+ Collections.sort(sorted);
+ return sorted;
+ }
+
+ private static long snapshotCount(Table table) {
+ table.refresh();
+ Map<Long, Snapshot> snapshots = new TreeMap<>();
+ table.snapshots().forEach(snapshot ->
snapshots.put(snapshot.snapshotId(), snapshot));
+ return snapshots.size();
+ }
+
+ private static SeaTunnelRow insert(long id, String name) {
+ return row(RowKind.INSERT, id, name);
+ }
+
+ private static SeaTunnelRow row(RowKind kind, long id, String name) {
+ SeaTunnelRow row = new SeaTunnelRow(new Object[] {id, name});
+ row.setRowKind(kind);
+ return row;
+ }
+
+ private static IcebergTableLoader loader(Table table) {
+ IcebergTableLoader loader = mock(IcebergTableLoader.class);
+ when(loader.loadTable()).thenReturn(table);
+ when(loader.getTableIdentifier()).thenReturn(TABLE);
+ return loader;
+ }
+
+ private static IcebergSinkConfig config(Long flushRowSize, boolean upsert,
String partitions) {
+ Map<String, Object> options = new HashMap<>();
+ if (flushRowSize != null) {
+ options.put(IcebergSinkOptions.FLUSH_ROW_SIZE.key(), flushRowSize);
+ }
+ if (upsert) {
+ options.put(IcebergSinkOptions.TABLE_PRIMARY_KEYS.key(), "id");
+
options.put(IcebergSinkOptions.TABLE_UPSERT_MODE_ENABLED_PROP.key(), true);
+ }
+ if (partitions != null && !partitions.isEmpty()) {
+ options.put(IcebergSinkOptions.TABLE_DEFAULT_PARTITION_KEYS.key(),
partitions);
+ }
+ return sinkConfig(options);
+ }
+
+ private static IcebergSinkConfig sinkConfig(Map<String, Object> options) {
+ Map<String, Object> config = new HashMap<>(options);
+ config.put(IcebergCommonOptions.KEY_CATALOG_NAME.key(), "test");
+ config.put(IcebergCommonOptions.CATALOG_PROPS.key(),
Collections.emptyMap());
+ return new IcebergSinkConfig(ReadonlyConfig.fromMap(config));
+ }
+
+ private static TableSchema idAndName() {
+ return TableSchema.builder()
+ .column(
+ PhysicalColumn.of(
+ "id", BasicType.LONG_TYPE, (Long) null, false,
null, null))
+ .column(
+ PhysicalColumn.of(
+ "name", BasicType.STRING_TYPE, (Long) null,
true, null, null))
+ .build();
+ }
+}
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/java/org/apache/seatunnel/e2e/connector/iceberg/IcebergSinkIT.java
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/java/org/apache/seatunnel/e2e/connector/iceberg/IcebergSinkIT.java
index 8be7e87df2..f89efea802 100644
---
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/java/org/apache/seatunnel/e2e/connector/iceberg/IcebergSinkIT.java
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/java/org/apache/seatunnel/e2e/connector/iceberg/IcebergSinkIT.java
@@ -21,6 +21,7 @@ import org.apache.seatunnel.api.configuration.ReadonlyConfig;
import org.apache.seatunnel.connectors.seatunnel.iceberg.IcebergTableLoader;
import
org.apache.seatunnel.connectors.seatunnel.iceberg.config.IcebergCommonOptions;
import
org.apache.seatunnel.connectors.seatunnel.iceberg.config.IcebergSourceConfig;
+import
org.apache.seatunnel.connectors.seatunnel.iceberg.sink.commit.IcebergFilesCommitter;
import
org.apache.seatunnel.connectors.seatunnel.iceberg.utils.HadoopStreamFileIO;
import org.apache.seatunnel.e2e.common.TestSuiteBase;
import org.apache.seatunnel.e2e.common.container.ContainerExtendedFactory;
@@ -34,6 +35,7 @@ import org.apache.seatunnel.e2e.common.util.DependencyJar;
import org.apache.iceberg.HasTableOperations;
import org.apache.iceberg.PartitionField;
import org.apache.iceberg.Schema;
+import org.apache.iceberg.Snapshot;
import org.apache.iceberg.Table;
import org.apache.iceberg.data.IcebergGenerics;
import org.apache.iceberg.data.Record;
@@ -56,6 +58,7 @@ import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.TreeMap;
import java.util.concurrent.TimeUnit;
import static
org.apache.seatunnel.connectors.seatunnel.iceberg.config.IcebergCatalogType.HADOOP;
@@ -74,6 +77,8 @@ public class IcebergSinkIT extends TestSuiteBase {
private static final String V3_COPY_TABLE = "iceberg_v3_copy";
private static final String V3_UPSERT_TABLE = "iceberg_v3_upsert";
private static final String V4_TABLE = "iceberg_v4_table";
+ private static final String FLUSH_UPSERT_TABLE =
"iceberg_flush_upsert_table";
+ private static final String FLUSH_CDC_TABLE = "iceberg_flush_cdc_table";
@TestContainerExtension
protected final ContainerExtendedFactory extendedFactory =
@@ -188,6 +193,109 @@ public class IcebergSinkIT extends TestSuiteBase {
"the job should report the unsupported format-version");
}
+ /**
+ * Upsert job with {@code flush_row_size = 2}: the UPDATE and DELETE of
earlier rows are written
+ * by later task writers, so they only apply if the checkpoint is
committed as ordered
+ * snapshots.
+ */
+ @TestTemplate
+ @DisabledOnContainer(
+ value = {},
+ type = {EngineType.SPARK},
+ disabledReason = "Spark does not carry the row kind of changelog
rows")
+ public void testFlushRowSizeUpsertE2e(TestContainer container)
+ throws IOException, InterruptedException {
+ assertFlushRowSizeJob(
+ container,
+ "/iceberg/fake_to_iceberg_flush_row_size_upsert.conf",
+ FLUSH_UPSERT_TABLE);
+ }
+
+ /** Same rows as the upsert case, written as a changelog into a
partitioned table. */
+ @TestTemplate
+ @DisabledOnContainer(
+ value = {},
+ type = {EngineType.SPARK},
+ disabledReason = "Spark does not carry the row kind of changelog
rows")
+ public void testFlushRowSizeCdcPartitionedE2e(TestContainer container)
+ throws IOException, InterruptedException {
+ assertFlushRowSizeJob(
+ container, "/iceberg/fake_to_iceberg_flush_row_size_cdc.conf",
FLUSH_CDC_TABLE);
+
Assertions.assertFalse(loadIcebergTableObject(FLUSH_CDC_TABLE).spec().isUnpartitioned());
+ }
+
+ private void assertFlushRowSizeJob(TestContainer container, String conf,
String tableName)
+ throws IOException, InterruptedException {
+ resetTableDirectory(container, tableName);
+
+ Container.ExecResult execResult = container.executeJob(conf);
+ Assertions.assertEquals(0, execResult.getExitCode(),
execResult.getStderr());
+
+ Table table = loadIcebergTableObject(tableName);
+ Map<Integer, String> rows = new TreeMap<>();
+ try (CloseableIterable<Record> records =
IcebergGenerics.read(table).build()) {
+ for (Record record : records) {
+ Assertions.assertNull(
+ rows.put((Integer) record.getField("id"), (String)
record.getField("name")),
+ "duplicate row for id " + record.getField("id"));
+ }
+ }
+ // 1 and 3 were updated and 2 deleted after the writer that inserted
them was rotated
+ Map<Integer, String> expected = new TreeMap<>();
+ expected.put(1, "A2");
+ expected.put(3, "C2");
+ expected.put(4, "D");
+ Assertions.assertEquals(expected, rows);
+
+ // The changes after a rotation must have been committed as ordered
snapshots, each group
+ // exactly once.
+ List<Snapshot> snapshots = new ArrayList<>();
+ table.snapshots().forEach(snapshots::add);
+ int splitCheckpoints = 0;
+ for (int i = 0; i < snapshots.size(); i++) {
+ String count =
+ snapshots
+ .get(i)
+ .summary()
+
.get(IcebergFilesCommitter.SNAPSHOT_PROPERTY_COMMIT_GROUP_COUNT);
+ if (count == null) {
+ continue;
+ }
+ int groupCount = Integer.parseInt(count);
+ Assertions.assertTrue(groupCount > 1, "a split checkpoint has at
least two groups");
+ Assertions.assertTrue(
+ i + groupCount <= snapshots.size(),
+ "missing groups of a split checkpoint: " +
summaries(snapshots));
+ String checkpointId =
+ snapshots
+ .get(i)
+ .summary()
+
.get(IcebergFilesCommitter.SNAPSHOT_PROPERTY_CHECKPOINT_ID);
+ for (int group = 0; group < groupCount; group++) {
+ Map<String, String> summary = snapshots.get(i +
group).summary();
+ Assertions.assertEquals(
+ String.valueOf(group),
+
summary.get(IcebergFilesCommitter.SNAPSHOT_PROPERTY_COMMIT_GROUP),
+ "groups must be committed once and in order: " +
summaries(snapshots));
+ Assertions.assertEquals(
+ checkpointId,
+
summary.get(IcebergFilesCommitter.SNAPSHOT_PROPERTY_CHECKPOINT_ID));
+ }
+ i += groupCount - 1;
+ splitCheckpoints++;
+ }
+ Assertions.assertTrue(
+ splitCheckpoints > 0,
+ "expected the checkpoint to be committed as ordered snapshots:
"
+ + summaries(snapshots));
+ }
+
+ private static List<Map<String, String>> summaries(List<Snapshot>
snapshots) {
+ List<Map<String, String>> summaries = new ArrayList<>();
+ snapshots.forEach(snapshot -> summaries.add(snapshot.summary()));
+ return summaries;
+ }
+
private void assertFormatV3Table(Table table, boolean nanosecondRows)
throws IOException {
Assertions.assertEquals(
3, ((HasTableOperations)
table).operations().current().formatVersion());
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/fake_to_iceberg_flush_row_size_cdc.conf
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/fake_to_iceberg_flush_row_size_cdc.conf
new file mode 100644
index 0000000000..c0f2cb2512
--- /dev/null
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/fake_to_iceberg_flush_row_size_cdc.conf
@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Changelog (CDC) job into a partitioned table with flush_row_size = 2, used
by IcebergSinkIT.
+# Same rows as fake_to_iceberg_flush_row_size_upsert.conf, but UPDATE_BEFORE
and DELETE are written
+# as deletes of the full row, and the writer rotation closes the files of
every open partition.
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ FakeSource {
+ plugin_output = "fake"
+ schema = {
+ fields {
+ id = int
+ name = string
+ region = string
+ }
+ }
+ rows = [
+ {kind = INSERT, fields = [1, "A", "east"]},
+ {kind = INSERT, fields = [2, "B", "west"]},
+ {kind = INSERT, fields = [3, "C", "east"]},
+ {kind = UPDATE_BEFORE, fields = [1, "A", "east"]},
+ {kind = UPDATE_AFTER, fields = [1, "A2", "east"]},
+ {kind = DELETE, fields = [2, "B", "west"]},
+ {kind = UPDATE_BEFORE, fields = [3, "C", "east"]},
+ {kind = UPDATE_AFTER, fields = [3, "C2", "east"]},
+ {kind = INSERT, fields = [4, "D", "west"]}
+ ]
+ }
+}
+
+sink {
+ Iceberg {
+ plugin_input = "fake"
+ catalog_name = "seatunnel_test"
+ iceberg.catalog.config = {
+ "type" = "hadoop"
+ "warehouse" = "file:///tmp/seatunnel_mnt/iceberg/hadoop-sink/"
+ }
+ namespace = "seatunnel_namespace"
+ table = "iceberg_flush_cdc_table"
+ iceberg.table.primary-keys = "id"
+ iceberg.table.partition-keys = "region"
+ flush_row_size = 2
+ case_sensitive = true
+ }
+}
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/fake_to_iceberg_flush_row_size_upsert.conf
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/fake_to_iceberg_flush_row_size_upsert.conf
new file mode 100644
index 0000000000..acb68c958e
--- /dev/null
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/fake_to_iceberg_flush_row_size_upsert.conf
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Upsert job with flush_row_size = 2, used by IcebergSinkIT. The sink writer
is rotated every
+# two rows, so the UPDATE and DELETE of rows 1, 2 and 3 are written by later
task writers than
+# the rows they change. All rows end up in one checkpoint, which must be
committed as ordered
+# snapshots for the equality deletes to apply to the earlier rows.
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ FakeSource {
+ plugin_output = "fake"
+ schema = {
+ fields {
+ id = int
+ name = string
+ region = string
+ }
+ }
+ rows = [
+ {kind = INSERT, fields = [1, "A", "east"]},
+ {kind = INSERT, fields = [2, "B", "west"]},
+ {kind = INSERT, fields = [3, "C", "east"]},
+ {kind = UPDATE_BEFORE, fields = [1, "A", "east"]},
+ {kind = UPDATE_AFTER, fields = [1, "A2", "east"]},
+ {kind = DELETE, fields = [2, "B", "west"]},
+ {kind = UPDATE_BEFORE, fields = [3, "C", "east"]},
+ {kind = UPDATE_AFTER, fields = [3, "C2", "east"]},
+ {kind = INSERT, fields = [4, "D", "west"]}
+ ]
+ }
+}
+
+sink {
+ Iceberg {
+ plugin_input = "fake"
+ catalog_name = "seatunnel_test"
+ iceberg.catalog.config = {
+ "type" = "hadoop"
+ "warehouse" = "file:///tmp/seatunnel_mnt/iceberg/hadoop-sink/"
+ }
+ namespace = "seatunnel_namespace"
+ table = "iceberg_flush_upsert_table"
+ iceberg.table.primary-keys = "id"
+ iceberg.table.upsert-mode-enabled = true
+ flush_row_size = 2
+ case_sensitive = true
+ }
+}