shunping commented on code in PR #40273:
URL: https://github.com/apache/beam/pull/40273#discussion_r4109603385
##########
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilV2.java:
##########
@@ -116,6 +126,14 @@ public GcsUtilV2 create(PipelineOptions options) {
/** Maximum number of requests permitted in a GCS batch request. */
private static final int MAX_REQUESTS_PER_BATCH = 100;
+ /**
+ * Upload chunk size applied when the pipeline does not ask for one. Mirrors
gcsio's {@code
+ * AsyncWriteChannelOptions} default, which java-storage does not share.
+ */
+ @VisibleForTesting
+ static final int DEFAULT_UPLOAD_CHUNK_SIZE_BYTES =
+ Runtime.getRuntime().maxMemory() < 512 * 1024 * 1024 ? 8 * 1024 * 1024 :
3 * 8 * 1024 * 1024;
+
Review Comment:
That's from the gcsio library which GcsUtil V1 depends on.
https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/e4afe16ab78a50774df05bb2ba62bbb18b34e0f2/util/src/main/java/com/google/cloud/hadoop/util/AsyncWriteChannelOptions.java#L72-L75
Here, we want to match the GcsUtil V2 behavior with V1, so we set the upload
buffer with the same logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
