mxm commented on PR #778:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/778#issuecomment-1954334343
Hi @1996fanrui! Let me answer your questions inline:
> Hi @mxm , I'm worried that this design for some reasons:
>
> 1. Some of memories cannot be detected. Such as: managed memroy. Flink
doen't know how many memories rocksdb is using. So the managed memeory usage
will be 100% once the slot is using rocksdb. (even if rocksdb only uses 1%
memory)
That is something I thought so as well, but it is not true. Here is a look
at a RocksDB configured job:
As you can see, not all the managed memory is used. You are right though
that RocksDB tends to inherit a lot of the managed memory. That is completely
in line with the design though. Managed memory is assigned last and the purpose
is to assign as much memory as possible to RocksDB, while assigning enough
memory to other memory pools beforehand and keeping the total memory within the
max specified TaskManager memory, never exceeding it.
> 2. `NETWORK_MEMORY_USED` may be unstable, Network memory usage will be
affected by many factors, such as:
>
> * The jobVertex parallelism
> * How many tasks are running in this TaskManager? To be honest, Flink
scheduler has some random factors when scheduling same job.
> * Some tasks can be finished. The network memory will be reduced after
some tasks are finished, then the network memory usage will be not accurate.
> * When the network memory isn't enough, the job will throw
`Insufficient number of network buffers`.
> * Our internal flink version designed a logic to calculate how many
network memory needed based on flink network configuration options and
`JobTopology`. It saves a lot of network memories, and flink streaming job
never throw `Insufficient number of network buffers`. I think I can help share
or contribute it in the next version. (I can draft a design doc recently.)
I'm open to improving this logic and curious about your ideas. The current
logic uses max usage + 20% (configurable) overhead which worked well in tests.
Also, we will never use less than the minimum configured limit (256MB by
default). I have an upcoming PR to scale network buffers alongside with scaling
decisions, based on the number of added/removed TaskManagers. In any case, we
can always add something more conservative for network buffers. You are right,
it may be better to be more conservative here to prevent failures.
> 3. AVG and MAX don't work for some memory area.
>
> * AutoScalerOptions#MEMORY_TUNING_TARGET has 2 options: AVG and MAX.
> * For current PR, all memories area shared same option.
> * AVG may works for heap memory. But the network/managed/ metaspace
only can use MAX, right? If the memory is not enough for the highest load TM,
it's easy to throw OOM exception.
That is right, I may remove this option entirely or restrict it to heap
only. It is already set to MAX by default.
> 4. `MEMORY_TUNING_MIN` is 256 MB for all types of memory. It will waste
some memories.
>
> * When the parallelism is too low , the needed network memory is too
less.
> * When the job without shuffle, the needed network memory 0 MB.
> * In general, the default metaspace memory is 256 MB, and most of flink
jobs only need a little(less than 256 MB)
> * When the streaming job without rocksdb, the managed memory can be 0MB
> * The heap memory for most of jobs are great than 256 MB.
> * So we set the same `MEMORY_TUNING_MIN= 256 M` for all type of
memories, we will waste some memories for managed/network/metaspace area.
True, I went the conservative route here for the first implementation.
Generally speaking, it should be safe to get rid of the minimum memory for
heap, metaspace, managed memory, and network buffers (if we calculate them
correctly). I suppose we could calculate the minimum network memory by using
this formula:
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/network_mem_tuning/#network-buffer-lifecycle
The savings will be huge with this change, 256MB makes hardly any difference
in our environment. I'd rather start with huge savings and optimize later. It
is clear that, without this change, there are no savings at all. Nor any
optimizations in terms of memory allocation of the different memory pools.
> In general, flink should be compatible with old options in subsequent
versions. The `flink-kubernetes-operator` will be release soon, I'm afraid we
release some unstable configuration options.
We can delist the memory tuning configuration options if you are concerned.
>
> Would you mind if we only polish heap memory tuning in this version?
Thanks~
Frankly yes, the heap-only solution won't provide much benefit. Memory
tuning is way more powerful if it manages all aspects of the memory
configuration. Improving heap memory is only a minor win for the configuration
work users face. We face so many issues daily with setting metaspace, network,
and managed memory correctly. Besides, the code is greatly improved over the
initial pull request. We can address any shortcomings here. The two main
concerns seem to be around (1) Setting network memory (2) Removing the minimum
memory constraint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
