Hi, For a complex streaming job, is there any way to tilt the memory towards > stateful operators?
If streaming jobs are interested, the quick answer is no. Memory is fetched on demand for all operators. Currently, only managed memory for batch jobs are pre-planned for each operator. Thank you~ Xintong Song On Mon, Aug 31, 2020 at 1:33 PM lec ssmi <shicheng31...@gmail.com> wrote: > HI: > Generally speaking, when we submitting the flink program, the number of > taskmanager and the memory of each tn will be specified. And the smallest > real execution unit of flink should be operator. > Since the calculation logic corresponding to each operator is > different, some need to save the state, and some don't. Therefore, the > memory size required by each operator should be different. How does the > flink program allocate taskmanager memory to the operator by default? > In our production practice, with the increase of traffic, some operators > (mainly stateful such as join and groupby) often have insufficient memory, > resulting in slower calculations. The usual approach is to increase the > entire taskmanager memory. But will this part of the increased memory be > allocated to the map-like operator, or that the memory itself is fetched on > demand in the same taskmanager whoever needs the memory will fetch it > until the memory is used up, in other words, there is no preset memory > allocation ratio. For a complex streaming job, is there any way to tilt the > memory towards stateful operators? > > Thanks. > > > >
