hi, Raúl (1)&(2) yes, the OS needs some pressure to release it. For example, if you have a total 16GB ram in your machine, then you read a file of 8GB and immediately close it. Noe the page cache would cache 8GB the file data. Then you start a program requesting memory from OS, the OS will release the page cache as long as your request goes beyond 8GB.
(3) I think you can configure your JVM with a maximum heap size of 14GB (-xmx) and leave 2GB memory for OS. you will have memory elasticity with this configuration. The JVM will increase memory allocation from OS as long as new objects are created, but it is bounded by 14GB which will not cause memory swapping. For example, if your application only needs 8GB memory, then the rest 8GB can be used for page cache, improving you IO performance. Otherwise, if your application needs 14GB memory, then the JVM will force OS to release almost all page cache. In this situation, your IO performance may not be good, but you can hold more data (e.g, RDD) in your application. Wei -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
