On Apr 1, 2010, at 5:04 PM, Vasilis Liaskovitis wrote: >> > > ok. Now, considering a map side space buffer and a sort buffer, do > both account for tenured space for both map and reduce JVMs? I 'd > think the map side buffer gets used and tenured for map tasks and the > sort space gets used and tenured for the reduce task during sort/merge > phase. Would both spaces really be used in both kinds of tasks? >
It is my understanding that a JVM used for a map won't also be used for a reduce. The JVM reuse runs multiple maps or reduces in one process but not across both. The mapper does the majority of the sorting, the reducer mostly merges pre-sorted data. Each kind of task tends to have a different memory footprint, dependent on the job and data. >> The maximum number of map and reduce tasks per node applies no matter how >> many jobs are running. > > RIght. But depending on your job scheduler, isn't it possible that you > may be swapping the different jobs' JVM space in and out of physical > memory while scheduling all the parallel jobs? Especially if nodes > don't have huge amounts of memory, this scenario sounds likely. > To be more precise, the max number of map and reduce tasks corresponds with the maximum number of active JVMs of each type at the same time. When a job finishes all of its tasks, the JVMs for it are killed. A new job gets new JVMs. Running concurrent jobs means that each job has some fraction of these JVM slots occupied. So, there should be no swapping different Jobs JVMs in and out of RAM. The same number of active JVM's exists for one large job as it does for 4 concurrent jobs. >> > > Back to a single job running and assuming all heap space being used, > what percentage of a node's memory would you leave for other functions > esp. disk cache? I currently only have 25% of memory (~4GB) for > non-heapJVM data; I guess there should be a sweet-spot, probably > dependent on the job I/O characteristics. > It will dependon the job, its I/O, and the OS tuning. But 25% to 33% of memory for system file cache has worked for me (remember, the nodes aren't just for tasks, but also for HDFS). A small amount of swap-out isn't bad, since the JVM's expire and never swap-in. > - Vasilis
