So I have Beam job that I want to run with Flink locally. Problem is, I get the following error:
> java.io.IOException: Insufficient number of network buffers: required 32, but only 24 available. The total number of network buffers is currently set to 32768 of 32768 bytes each. You can increase this number by setting the configuration keys 'taskmanager.network.memory.fraction', 'taskmanager.network.memory.min', and 'taskmanager.network.memory.max'. So I create a config file with taskmanager.network.memory.max set to 5gb and taskmanager.network.memory.fraction set to 0.2. I also set the FLINK_CONF_DIR path to the dir with the config file (undocumented feature) and set the --flinkMaster path to "[local]" as it seems like the default "[auto]" ignores the config file: https://github.com/apache/beam/blob/1e41220977d6c45d293b86f2e581daec3513c66e/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L76-L82 . Now, it seems like configs are loaded ok. I get the following log message during start: > Jan 09, 2019 10:54:43 AM org.apache.flink.configuration.GlobalConfiguration loadYAMLResource INFO: Loading configuration property: taskmanager.network.memory.max, 5gb But the error at the top of the post still appears. 32768 * 32768 bytes = 1gb, which is the default value of taskmanager.network.memory.max, so it seems like the config is ignored. Any ideas what might cause this problem? Am I adjusting the wrong parameter or something? >
