DanielLeens commented on code in PR #13268:
URL: https://github.com/apache/seatunnel/pull/13268#discussion_r4565767946


##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java:
##########
@@ -260,8 +317,156 @@ public MultiTableSinkWriter(
             runnable.add(r);
         }
         log.info("init multi table sink writer, queue size: {}", queueSize);
-        initResourceManager(queueSize);
-        registerInitialFailedTables(initialFailedTables);
+        try {
+            initResourceManager(queueSize);
+            registerInitialFailedTables(initialFailedTables);
+        } catch (RuntimeException | Error e) {
+            executorService.shutdownNow();
+            closeWritersAfterInitFailure(sinkWriters.values(), e);
+            if (resourceManager != null) {
+                try {
+                    resourceManager.close();
+                } catch (Throwable closeError) {
+                    e.addSuppressed(closeError);
+                }
+            }
+            throw e;
+        }
+    }
+
+    /**
+     * Checks that the sub-writers can run with {@code replicaNum} writers of 
each table in one
+     * subtask. With one replica every mode is accepted. With more, each table 
needs {@link
+     * MultiTableSinkReplicaMode#KEYED} with a present, non-negative {@code 
primaryKey()}, or {@link
+     * MultiTableSinkReplicaMode#RANDOM}, and all writers of a table must 
declare the same mode and,
+     * for {@code KEYED}, the same key index.
+     *
+     * <p>It runs before any queue worker starts, for created and restored 
writers alike. A path
+     * that adds writers after construction must call it for the new writers 
before it registers
+     * them, so a table cannot start routing without a valid declaration.
+     *
+     * @param writers the sub-writers, keyed by table identifier and replica 
index
+     * @param replicaNum the value of {@code multi_table_sink_replica}
+     * @param connectorNames connector name of each table identifier, used in 
error messages
+     * @throws IllegalArgumentException if a table cannot run with {@code 
replicaNum} replicas; the
+     *     message names every such table, its connector and writer class
+     */
+    static void validateReplicaRouting(
+            Map<SinkIdentifier, SinkWriter<SeaTunnelRow, ?, ?>> writers,
+            int replicaNum,
+            Map<String, String> connectorNames) {
+        if (replicaNum <= 1) {
+            return;
+        }
+        Map<String, List<SinkWriter<SeaTunnelRow, ?, ?>>> writersByTable = new 
TreeMap<>();
+        writers.forEach(
+                (identifier, writer) ->
+                        writersByTable
+                                .computeIfAbsent(
+                                        identifier.getTableIdentifier(), key 
-> new ArrayList<>())
+                                .add(writer));
+        List<String> rejected = new ArrayList<>();
+        for (Map.Entry<String, List<SinkWriter<SeaTunnelRow, ?, ?>>> table :
+                writersByTable.entrySet()) {
+            MultiTableSinkReplicaMode mode = null;
+            Optional<Integer> key = Optional.empty();
+            String problem = null;
+            for (SinkWriter<SeaTunnelRow, ?, ?> writer : table.getValue()) {
+                MultiTableSinkReplicaMode writerMode = replicaModeOf(writer);
+                Optional<Integer> writerKey =
+                        writerMode == MultiTableSinkReplicaMode.KEYED
+                                ? primaryKeyOf(writer)
+                                : Optional.empty();
+                if (mode == null) {
+                    mode = writerMode;
+                    key = writerKey;
+                } else if (mode != writerMode || !key.equals(writerKey)) {
+                    problem =
+                            String.format(
+                                    "its writers do not agree: one declares 
%s, another %s",
+                                    describeMode(mode, key), 
describeMode(writerMode, writerKey));
+                    break;
+                }
+            }
+            if (problem == null && mode == 
MultiTableSinkReplicaMode.UNSUPPORTED) {
+                problem = "its writer declares neither KEYED nor RANDOM";
+            } else if (problem == null
+                    && mode == MultiTableSinkReplicaMode.KEYED
+                    && (!key.isPresent() || key.get() < 0)) {
+                problem =
+                        "its writer declares KEYED, but primaryKey() returns "
+                                + (key.isPresent() ? "index " + key.get() : 
"no key");
+            }
+            if (problem != null) {
+                String connector = connectorNames.get(table.getKey());
+                rejected.add(
+                        String.format(
+                                "table '%s' (%swriter %s): %s",
+                                table.getKey(),
+                                connector == null ? "" : "connector '" + 
connector + "', ",
+                                table.getValue().get(0).getClass().getName(),
+                                problem));
+            }
+        }
+        if (!rejected.isEmpty()) {

Review Comment:
   Thanks for asking. That exception comes from the shared writer, and only 
applies when more than one replica is requested for a table. The check is there 
because the queue assignment has to match what the sink does with the row. 
Iceberg's delta writer can delete and replace a row by its identifier. If an 
update and delete for the same record go to different queues, one writer can 
commit a stale value after the other has deleted it. The append writer has no 
such ordering dependency, so random routing is allowed there.
   
   The Iceberg-specific decision is in `IcebergSinkWriter.replicaMode()` and 
`primaryKey()`, while `MultiTableSinkWriter.validateReplicaRouting()` enforces 
the common contract. If the delta writer cannot identify a routing key, one 
replica is the safe fallback; the shared guard should not reject an append-only 
writer that declares RANDOM. The routing and recovery findings in my existing 
source review still apply. Also, the previously pending Apache 
[Build](https://github.com/apache/seatunnel/runs/131802756554) now reports 
`TIMED_OUT` and the exact-head [fork 
Build](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/43267729246)
 was cancelled; neither result clears those source findings.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to