Re: Question about cache
Hi Kuli Is Just raising. Thanks for the explanation. Regards Anderson 2012/5/11 Shawn Heisey s...@elyograg.org On 5/11/2012 9:30 AM, Anderson vasconcelos wrote: HI Kuli The free -m command gives me total used free sharedbuffers cached Mem: 9991 9934 57 0 75 5759 -/+ buffers/cache: 4099 5892 Swap: 8189 3395 4793 You can see that has only 57m free and 5GB cached. In top command, the glassfish process used 79,7% of memory: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 4336 root 21 0 29.7g 7.8g 4.0g S 0.3 79.7 5349:14 java If i increase the memory of server for more 2GB, the SO will be use this additional 2GB in cache? I need to increse the memory size? Are you having a problem you need to track down, or are you just raising a concern because your memory usage is not what you expected? It is 100% normal for a Linux system to show only a few megabytes of memory free. To make things run faster, the OS caches disk data using memory that is not directly allocated to programs or the OS itself. If a program requests memory, the OS will allocate it immediately, it simply forgets the least used part of the cache. Windows does this too, but Microsoft decided that novice users would freak out if the task manager were to give users the true picture of memory usage, so they exclude disk cache when calculating free memory. It's not really a lie, just not the full true picture. A recent version of Solr (3.5, if I remember right) made a major change in the way that the index files are accessed. The way things are done now is almost always faster, but it makes the memory usage in the top command completely useless. The VIRT memory size includes all of your index files, plus all the memory that the java process is capable of allocating, plus a little that i can't quite account for. The RES size is also bigger than expected, and I'm not sure why. Based on the numbers above, I am guessing that your indexes take up 15-20GB of disk space. For best performance, you would want a machine with at least 24GB of RAM so that your entire index can fit into the OS disk cache. The 10GB you have (which leaves the 5.8 GB for disk cache as you have seen) may be good enough to cache the frequently accessed portions of your index, so your performance might be just fine. Thanks, Shawn
Re: Question about cache
Am 11.05.2012 15:48, schrieb Anderson vasconcelos: Hi Analysing the solr server in glassfish with Jconsole, the Heap Memory Usage don't use more than 4 GB. But, when was executed the TOP comand, the free memory in Operating system is only 200 MB. The physical memory is only 10GB. Why machine used so much memory? The cache fields are included in Heap Memory usage? The other 5,8 GB is the caching of Operating System for recent open files? Exists some way to tunning this? Thanks If the OS is Linux or some other Unix variant, it keeps as much disk content in memory as possible. Whenever new memory is needed, it automatically gets freed. That won't need time, and there's no need to tune anything. Don't look at the free memory in top command, it's nearly useless. Have a look at how much memory your Glassfish process is consuming, and use the 'free' command (maybe together with the -m parameter for human readability) to find out more about your free memory. The -/+ buffers/cache line is relevant. Greetings, Kuli
Re: Question about cache
HI Kuli The free -m command gives me total used free sharedbuffers cached Mem: 9991 9934 57 0 75 5759 -/+ buffers/cache: 4099 5892 Swap: 8189 3395 4793 You can see that has only 57m free and 5GB cached. In top command, the glassfish process used 79,7% of memory: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 4336 root 21 0 29.7g 7.8g 4.0g S 0.3 79.7 5349:14 java If i increase the memory of server for more 2GB, the SO will be use this additional 2GB in cache? I need to increse the memory size? Thanks 2012/5/11 Michael Kuhlmann k...@solarier.de Am 11.05.2012 15:48, schrieb Anderson vasconcelos: Hi Analysing the solr server in glassfish with Jconsole, the Heap Memory Usage don't use more than 4 GB. But, when was executed the TOP comand, the free memory in Operating system is only 200 MB. The physical memory is only 10GB. Why machine used so much memory? The cache fields are included in Heap Memory usage? The other 5,8 GB is the caching of Operating System for recent open files? Exists some way to tunning this? Thanks If the OS is Linux or some other Unix variant, it keeps as much disk content in memory as possible. Whenever new memory is needed, it automatically gets freed. That won't need time, and there's no need to tune anything. Don't look at the free memory in top command, it's nearly useless. Have a look at how much memory your Glassfish process is consuming, and use the 'free' command (maybe together with the -m parameter for human readability) to find out more about your free memory. The -/+ buffers/cache line is relevant. Greetings, Kuli
Re: Question about cache
On 5/11/2012 9:30 AM, Anderson vasconcelos wrote: HI Kuli The free -m command gives me total used free sharedbuffers cached Mem: 9991 9934 57 0 75 5759 -/+ buffers/cache: 4099 5892 Swap: 8189 3395 4793 You can see that has only 57m free and 5GB cached. In top command, the glassfish process used 79,7% of memory: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 4336 root 21 0 29.7g 7.8g 4.0g S 0.3 79.7 5349:14 java If i increase the memory of server for more 2GB, the SO will be use this additional 2GB in cache? I need to increse the memory size? Are you having a problem you need to track down, or are you just raising a concern because your memory usage is not what you expected? It is 100% normal for a Linux system to show only a few megabytes of memory free. To make things run faster, the OS caches disk data using memory that is not directly allocated to programs or the OS itself. If a program requests memory, the OS will allocate it immediately, it simply forgets the least used part of the cache. Windows does this too, but Microsoft decided that novice users would freak out if the task manager were to give users the true picture of memory usage, so they exclude disk cache when calculating free memory. It's not really a lie, just not the full true picture. A recent version of Solr (3.5, if I remember right) made a major change in the way that the index files are accessed. The way things are done now is almost always faster, but it makes the memory usage in the top command completely useless. The VIRT memory size includes all of your index files, plus all the memory that the java process is capable of allocating, plus a little that i can't quite account for. The RES size is also bigger than expected, and I'm not sure why. Based on the numbers above, I am guessing that your indexes take up 15-20GB of disk space. For best performance, you would want a machine with at least 24GB of RAM so that your entire index can fit into the OS disk cache. The 10GB you have (which leaves the 5.8 GB for disk cache as you have seen) may be good enough to cache the frequently accessed portions of your index, so your performance might be just fine. Thanks, Shawn
