LeonYoah commented on code in PR #11746:
URL: https://github.com/apache/seatunnel/pull/11746#discussion_r3764122515
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/DynamicChunkSplitter.java:
##########
@@ -106,6 +115,324 @@ private Collection<JdbcSourceSplit> createDynamicSplits(
return splits;
}
+ private static final String COMPOSITE_KEY_SEPARATOR = ",";
+
+ /**
+ * Split a table on the full composite key tuple. Boundaries are
tuple-ordered ((start, end] per
+ * chunk, matching {@link #buildCompositeCondition}), so composite-key
tables whose first column
+ * repeats heavily still split into balanced chunks via the remaining key
columns.
+ *
+ * <p><b>Performance trade-off:</b> unlike the single-column paths which
compute chunk
+ * boundaries arithmetically (closed-form evenly/unevenly sized chunks),
this path walks the
+ * boundary with one database round trip per chunk ({@link
#queryNextChunkMaxComposite} scans at
+ * most {@code split.size} rows per call). For a large table with many
chunks this is more round
+ * trips than the single-column arithmetic path, in exchange for correct,
balanced chunks on
+ * composite keys where arithmetic is impossible (the first column has low
cardinality).
+ * Boundary queries are index-friendly (expanded OR/AND), so each round
trip is cheap.
+ */
+ private Collection<JdbcSourceSplit> createCompositeDynamicSplits(
+ JdbcSourceTable table, SeaTunnelRowType splitKey) throws
SQLException {
+ String[] columns = splitKey.getFieldNames();
+ Object[][] minMax = queryMinMaxComposite(table, columns);
+ Object[] min = minMax[0];
+ Object[] max = minMax[1];
+ if (min == null || max == null) {
+ // empty table, return a single full-table split
+ return Collections.singletonList(createCompositeSplit(table, 0,
splitKey, null, null));
+ }
+
+ // first chunk end
+ Object[] firstEnd = queryNextChunkMaxComposite(table, columns,
config.getSplitSize(), min);
Review Comment:
I think this approach has performance implications. To generate boundaries,
it will traverse and transfer almost all primary keys in batches before the
task actually reads the data, and then the reader will read the entire table
data again. For large tables, the startup time, network overhead, and temporary
object allocation can be significant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
