Hi Roman, Thanks for the proposal, this allows State Backend to make better use of memory.
After reading the ticket, I'm curious about some points: 1. Is shared-memory only for the state backend? If both "taskmanager.memory.managed.shared-fraction: >0" and "state.backend.rocksdb.memory.managed: false" are set at the same time, will the shared-memory be wasted? 2. It's said that "Jobs 4 and 5 will use the same 750Mb of unmanaged memory and will compete with each other" in the example, how is the memory size of unmanaged part calculated? 3. For fine-grained-resource-management, the control of cpuCores, taskHeapMemory can still work, right? And I am a little worried that too many memory-about configuration options are complicated for users to understand. Regards, Yanfei Roman Khachatryan <ro...@apache.org> 于2022年11月8日周二 23:22写道： > Hi everyone, > > I'd like to discuss sharing RocksDB memory across slots as proposed in > FLINK-29928 [1]. > > Since 1.10 / FLINK-7289 [2], it is possible to: > - share these objects among RocksDB instances of the same slot > - bound the total memory usage by all RocksDB instances of a TM > > However, the memory is divided between the slots equally (unless using > fine-grained resource control). This is sub-optimal if some slots contain > more memory intensive tasks than the others. > Using fine-grained resource control is also often not an option because the > workload might not be known in advance. > > The proposal is to widen the scope of sharing memory to TM, so that it can > be shared across all RocksDB instances of that TM. That would reduce the > overall memory consumption in exchange for resource isolation. > > Please see FLINK-29928 [1] for more details. > > Looking forward to feedback on that proposal. > > [1] > https://issues.apache.org/jira/browse/FLINK-29928 > [2] > https://issues.apache.org/jira/browse/FLINK-7289 > > Regards, > Roman >
