Stephan Ewen created FLINK-23093:
------------------------------------
Summary: Limit number of I/O pool and Future threads in Mini
Cluster
Key: FLINK-23093
URL: https://issues.apache.org/jira/browse/FLINK-23093
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Stephan Ewen
Fix For: 1.14.0
When running tests on CI via the minicluster, the mini cluster typically spawns
100s of I/O threads, both in the MiniCluster I/O pool and in the TM I/O pool.
The standard rule for the maximum pool size is 4*num-cores, but the number of
cores can be fairly large these days. Various Java versions also mess up core
counting when running in containers (JVM container might have been given 2
cores as resource limits, but the JVM counts the system as a whole, like 64/128
cores).
This is both a nuisance for debugging, and a big waste of memory (each thread
takes by default around 1MB when spawned, so the test JVM wastes 100s of MBs
for nothing).
I would suggest to set a default of 8 I/O threads for the Mini Cluster. The
scaling-with-cores is important for proper TM/JM deployments, but not for the
Mini Cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
