Hi All, We are trying to running our job in cluster that has this information
1. # of machine: 16 2. memory : 128 gb 3. # of core : 48 However when we try to run we have an exception. "insufficient number of network buffers. 48 required but only 10 available. the total number of network buffers is currently set to 2048" After looking at the documentation we set configuration based on docs taskmanager.network.numberOfBuffers: # core ^ 2 * # machine * 4 However we face another error from JVM java.io.IOException: Cannot allocate network buffer pool: Could not allocate enough memory segments for NetworkBufferPool (required (Mb): 2304, allocated (Mb): 698, missing (Mb): 1606). Cause: Java heap space We fiddle the taskmanager.heap.mb: 4096 Finally the cluster is running. However i'm still not sure about the configuration and fiddling in task manager heap really fine tune. So my question is 1. Am i doing it right for numberOfBuffers ? 2. How much should we allocate on taskmanager.heap.mb given the information 3. Any suggestion which configuration we need to set to make it optimal for the cluster ? 4. Is there any chance that this will get automatically resolve by memory/network buffer manager ? Thanks a lot for the help Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>
