li3zhi4 commented on code in PR #11746:
URL: https://github.com/apache/seatunnel/pull/11746#discussion_r3766447820
##########
seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/DynamicChunkSplitter.java:
##########
@@ -106,6 +115,324 @@ private Collection<JdbcSourceSplit> createDynamicSplits(
return splits;
}
+ private static final String COMPOSITE_KEY_SEPARATOR = ",";
+
+ /**
+ * Split a table on the full composite key tuple. Boundaries are
tuple-ordered ((start, end] per
+ * chunk, matching {@link #buildCompositeCondition}), so composite-key
tables whose first column
+ * repeats heavily still split into balanced chunks via the remaining key
columns.
+ *
+ * <p><b>Performance trade-off:</b> unlike the single-column paths which
compute chunk
+ * boundaries arithmetically (closed-form evenly/unevenly sized chunks),
this path walks the
+ * boundary with one database round trip per chunk ({@link
#queryNextChunkMaxComposite} scans at
+ * most {@code split.size} rows per call). For a large table with many
chunks this is more round
+ * trips than the single-column arithmetic path, in exchange for correct,
balanced chunks on
+ * composite keys where arithmetic is impossible (the first column has low
cardinality).
+ * Boundary queries are index-friendly (expanded OR/AND), so each round
trip is cheap.
+ */
+ private Collection<JdbcSourceSplit> createCompositeDynamicSplits(
+ JdbcSourceTable table, SeaTunnelRowType splitKey) throws
SQLException {
+ String[] columns = splitKey.getFieldNames();
+ Object[][] minMax = queryMinMaxComposite(table, columns);
+ Object[] min = minMax[0];
+ Object[] max = minMax[1];
+ if (min == null || max == null) {
+ // empty table, return a single full-table split
+ return Collections.singletonList(createCompositeSplit(table, 0,
splitKey, null, null));
+ }
+
+ // first chunk end
+ Object[] firstEnd = queryNextChunkMaxComposite(table, columns,
config.getSplitSize(), min);
Review Comment:
One more clarification on the startup-time impact, since I want to be
upfront about it: for tables that previously took the single-column arithmetic
path (O(1) split analysis), switching to the composite seek does lengthen the
split-preparation phase - the boundary walk is O(N) in total before any reader
starts. We are aware of this trade-off and have a plan to address it: streaming
boundary generation, i.e. producing and handing off the next boundary/split to
the downstream readers while they consume the current chunk, so the
split-preparation work overlaps with reading instead of blocking startup. This
is a follow-up change we intend to track separately (it touches
split-generation timing, orthogonal to the composite-key selection itself).
Happy to discuss the design if you would like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
