hudi-agent commented on code in PR #19550:
URL: https://github.com/apache/hudi/pull/19550#discussion_r3735140343
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/SparkSampleWritesUtils.java:
##########
@@ -62,6 +65,24 @@
@Slf4j
public class SparkSampleWritesUtils {
+ /**
+ * The sampled records are shipped to the executor inside a single Spark task
+ * ({@code jsc.parallelize(samples, 1)}). If their total serialized size
approaches the RPC frame
+ * limit, launching that task fails with "exceeds max allowed:
spark.rpc.message.maxSize". We
+ * therefore cap the sample at this fraction of {@code
spark.rpc.message.maxSize}; the remaining
+ * headroom absorbs the task closure, RDD metadata, and the difference
between the Kryo estimate
+ * used here and the serializer Spark actually uses when shipping the task.
+ */
+ private static final double SAMPLE_WRITES_TASK_BYTES_FRACTION = 0.5;
+
+ /** Spark config bounding the maximum RPC frame size; a serialized task may
not exceed it. */
+ private static final String RPC_MESSAGE_MAX_SIZE_CONF =
"spark.rpc.message.maxSize";
+
+ /** Spark's default value for {@link #RPC_MESSAGE_MAX_SIZE_CONF}, in MiB
(128 MiB). */
Review Comment:
🤖 nit: 1048576 bytes is one mebibyte (MiB), not one megabyte (MB = 1 000 000
bytes), and the Javadoc on the field directly above already says "in MiB".
Could you rename this to `BYTES_PER_MIB` so the constant name stays consistent
with the surrounding docs?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
