Hey Sebastian, did you already try to increase the number of buffers in accordance to Stephan's suggestion? The current defaults for the number and size of network buffers are 2048 and 32768 bytes, resulting in 64 MB of memory for the network buffers.
Out of curiosity: on how many machines are you running your job and what parallelism did you set for your program? Best, Ufuk On 04 Jul 2014, at 15:46, Kruse, Sebastian <sebastian.kr...@hpi.de> wrote: > Hi everyone, > > I apologize in advance if that is not the right mailing list for my question. > If there is a better place for it, please let me know. > > Basically, I wanted to ask if you have some statement about the hardware > requirements of Flink to process larger amounts of data beginning from, say, > 20 GBs. Currently, I am facing issues in my jobs, e.g., there are not enough > buffers for safe execution of some operations. Since the machines that run my > TaskTrackers have unfortunately very limited main memory, I cannot increase > the number of buffers (and heap space in general) too much. Currently, I > assigned them 1.5 GB. > > So, the exact questions are: > > * Do you have experiences with a suitable HW setup for crunching > larger amounts of data, maybe from the TU cluster? > > * Are there any configuration tips, you can provide, e.g. pertaining > to the buffer configuration? > > * Are there any general statements on the growth of Flink's memory > requirements wrt. to the size of the input data? > > Thanks for your help! > Sebastian
