Weijie Guo created FLINK-29425:
----------------------------------
Summary: Hybrid full spilling strategy triggering spilling
frequently
Key: FLINK-29425
URL: https://issues.apache.org/jira/browse/FLINK-29425
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.16.0
Reporter: Weijie Guo
In hybrid shuffle mode, we have an internal config option
'DEFAULT_FULL_STRATEGY_NUM_BUFFERS_TRIGGER_SPILLED' to control spilling
frequency in the full spilling strategy. Unfortunately, the default value(10)
is too small. As a result, frequent disk spilling calls are observed in the
TPC-DS test, which seriously affects performance. When we increase the value,
the query performance is improved significantly. We should set a more
reasonable default value, or adopt an adaptive strategy to determine the
spilling frequency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
