github-actions[bot] commented on code in PR #64511:
URL: https://github.com/apache/doris/pull/64511#discussion_r3419822092
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/ConfigUtil.java:
##########
@@ -123,9 +124,35 @@ public static ZoneId
getPostgresServerTimeZoneFromProps(java.util.Properties pro
return ZoneId.systemDefault();
}
+ public static final String MAX_QUEUE_BYTES_SYS_PROP =
"cdc.max.queue.size.in.bytes";
+
+ // Heap-adaptive byte cap for the debezium ChangeEventQueue buffer.
+ // heap 1G->64MB, 2G->128MB, >=4G->256MB. -D<MAX_QUEUE_BYTES_SYS_PROP>
overrides
+ // (<=0 disables); a malformed override is logged and ignored, falling
back to the cap.
+ private static long resolveMaxQueueSizeInBytes() {
+ String override = System.getProperty(MAX_QUEUE_BYTES_SYS_PROP);
+ if (override != null) {
+ try {
+ long bytes = Long.parseLong(override.trim());
+ return bytes <= 0 ? 0 : bytes;
+ } catch (NumberFormatException e) {
+ LOG.warn(
+ "Ignoring invalid -D{}={}, expected an integer byte
count; "
+ + "falling back to the adaptive cap",
+ MAX_QUEUE_BYTES_SYS_PROP,
+ override);
+ }
+ }
+ long target = Runtime.getRuntime().maxMemory() / 16;
+ return Math.max(64L * 1024 * 1024, Math.min(target, 256L * 1024 *
1024));
+ }
+
/** Optimized debezium parameters */
public static Properties getDefaultDebeziumProps() {
Properties properties = new Properties();
+ properties.put(
Review Comment:
Enabling this Debezium property activates `ObjectSizeCalculator` in every
reader execution, but the Java 17 module opens added in `CdcClientMgr` only
apply when BE forks `cdc-client.jar`. The current `Test Cdc Client` check runs
`mvn -B test failsafe:integration-test failsafe:verify` directly, so it does
not get those flags and the integration suite now fails with
`InaccessibleObjectException: Unable to make field transient
java.util.HashMap$Node[] java.util.HashMap.table accessible: module java.base
does not "opens java.util"`. Please add the same `--add-opens` set to the
cdc_client Surefire/Failsafe JVM, and to any supported non-BE startup path, or
avoid enabling byte sizing unless those opens are guaranteed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
