taoran92 opened a new issue, #10765: URL: https://github.com/apache/paimon/issues/10765
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Paimon supports managed writer buffers through `sink.use-managed-memory-allocator` and the allocation weight `sink.managed.writer-buffer-memory`. With default slot resources, Flink can already allocate managed memory to the writer using these settings. However, when users explicitly configure the writer slot sharing group with `sink.writer-memory`, Paimon specifies CPU and task heap resources without an option to specify managed memory. In Flink 1.20.1, the resulting explicit group resource profile contains zero managed memory. Changing the writer's allocation weight does not change this resource request. For production deployments that honor fine-grained slot resource requests, users may need, for example, 512 MiB of task heap for writer objects and encoding, and a 1 GiB managed-memory budget for the writer slot sharing group. Today, this combination cannot be expressed through the writer resource options. Users can use default slot resources instead, but that does not provide the same control over the writer group's resource requirements. An explicit managed budget would let users size heap and managed resources independently and express different ingestion workloads' buffering requirements to Flink's scheduler. ### Solution Add an optional `sink.writer-managed-memory` table property with no default value. When specified, set the writer slot sharing group's managed resource request through `SlotSharingGroup.Builder.setManagedMemory()`. Example: ```properties sink.writer-memory = 512 mb sink.writer-managed-memory = 128 mb sink.use-managed-memory-allocator = true sink.managed.writer-buffer-memory = 256 mb ``` A JobGraph comparison verified with Flink 1.20.1 shows the effect of adding the new option to the configuration above: ```text Before (without sink.writer-managed-memory): Writer SSG resource profile: task heap = 512 MiB, managed = 0 MiB Writer OPERATOR weight: 256 After (sink.writer-managed-memory = 128 mb): Writer SSG resource profile: task heap = 512 MiB, managed = 128 MiB Writer OPERATOR weight: 256 ``` These values describe the resource request, not the actual runtime buffer capacity. The new option specifies the managed memory resource request for each shared slot of the writer slot sharing group. It is not a total budget across all writer subtasks or a guaranteed buffer size for each writer. The existing allocator option selects the writer's allocation path, and `sink.managed.writer-buffer-memory` remains a weight. Flink computes each writer subtask's buffer capacity from the slot's actual managed memory and the use-case/operator fractions. In this example, `256 mb` is neither a fixed allocation nor a 256 MiB buffer limit. Require `sink.writer-memory` when the new option is specified, since a resourceful Flink slot sharing group requires CPU and task heap specifications. Require a positive managed budget whenever the new option is specified. Do not require the allocator switch merely to declare group resources. Preserve existing behavior when the new option is absent. Keep the existing downstream slot sharing behavior and committer resource configuration unchanged. Task off-heap settings and independent precommit compaction resources are outside this proposal. ### Anything else? Related capabilities: - https://github.com/apache/paimon/pull/1240: managed writer buffers and allocation weights. - https://github.com/apache/paimon/pull/5698: writer CPU and heap resource configuration. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
