taoran92 opened a new pull request, #10767:
URL: https://github.com/apache/paimon/pull/10767
### Purpose
Closes #10765.
When `sink.writer-memory` configures an explicit writer slot sharing group,
the group currently requests zero managed memory. Changing
`sink.managed.writer-buffer-memory` only changes the operator's allocation
weight, so users cannot request heap and managed resources independently.
Add the optional `sink.writer-managed-memory` property for ordinary writers
and the CDC combined sink. It requires `sink.writer-memory` and a positive
value, and sets the managed resource request for each shared slot of the writer
group. For example, a writer group can request 512 MiB of task heap and 128 MiB
of managed memory while retaining an operator weight of 256.
The option does not enable the managed allocator or change allocation
weights. Existing behavior when the option is absent, downstream slot sharing,
and explicit committer resources are preserved. Update the configuration guide
and generated option reference to explain these boundaries.
### Tests
Passed 33 tests across `WriterManagedMemoryTest`,
`FlinkCdcMultiTableSinkTest`, `FlinkSinkTest`, and
`WriterChainingStrategyTest`, with Checkstyle, Spotless, and Enforcer enabled:
```shell
mvn -o -pl paimon-flink/paimon-flink-cdc -am -Pflink1 \
-Dtest=WriterManagedMemoryTest,FlinkCdcMultiTableSinkTest,FlinkSinkTest,WriterChainingStrategyTest
\
-DwildcardSuites=none -DfailIfNoTests=false test
```
The tests cover StreamGraph and JobGraph resources, allocator enabled and
disabled, configuration validation, legacy behavior, and explicit committer
resources. Append and primary-key writers also run in a Flink operator test
harness using a 4 MiB managed budget read from JobGraph, checking buffer
capacity, usage, and commit output.
An independent JobGraph probe on Flink 1.20.1 verified that adding the new
option changes managed memory from 0 to 128 MiB while heap remains 512 MiB and
the operator weight remains 256. It also checked the default CPU value,
allocator-disabled configuration, invalid budgets, and default-slot behavior.
Production scheduler allocation was not tested.
`git diff --check` passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
