StephanEwen commented on pull request #13595: URL: https://github.com/apache/flink/pull/13595#issuecomment-717204918
@wsry I have two more questions on the minimum buffers logic and confi (1) I previously thought that this is to ensure that the sort buffers are not too small. But I saw now that they can actually be smaller, because the minimum is `Math.min(minBuffersPerSortMergePartition , numSubpartitions + 1)`. So if we have a low parallelism, then the sort shuffle might run with something like 5 buffers (assuming a user turns it on for all shuffles). Is that on purpose? Does it make sense to just use the `minBuffersPerSortMergePartition` value and not lower it below that? BTW: I think I now understand why you called it "max buffers" before - because it is the maximum to which the minimum requirement can grow. (2) The current value for `taskmanager.network.sort-shuffle.min-buffers` may be high for mini clusters. At least if we end up doing what is proposed in (1) and not lower it for lower parallelism. The current min is 512 buffers (16 MB) and the mini cluster has only 64 MB network memory by default. So as soon as there are four shuffles, the mini cluster will break, which is pretty soon. We should probably set a different min on the mini cluster, like 64 buffers or so. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
