Re: The VFS cache is not freed when there is not enough free memory to allocate
On 12/1/06, Nick Piggin <[EMAIL PROTECTED]> wrote: The pattern you are seeing here is probably due to the page allocator always retrying process context allocations which are <= order 3 (64K with 4K pages). You might be able to increase this limit a bit for your system, but it could easily cause problems. Especially fragmentation on nommu systems where the anonymous memory cannot be paged out. Thanks for your clue. I found increasing this limit could really help my test cases. When MemFree < 8M, and the test case request 1M * 8 times, the allocation can be sucessful after 81 times rebalance, :). So far I haven't found any issue. If I make a patch to move this parameter to be tunable in the proc filesystem on nommu case, is it acceptable? Thanks, -Aubrey - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
On 12/1/06, Nick Piggin [EMAIL PROTECTED] wrote: The pattern you are seeing here is probably due to the page allocator always retrying process context allocations which are = order 3 (64K with 4K pages). You might be able to increase this limit a bit for your system, but it could easily cause problems. Especially fragmentation on nommu systems where the anonymous memory cannot be paged out. Thanks for your clue. I found increasing this limit could really help my test cases. When MemFree 8M, and the test case request 1M * 8 times, the allocation can be sucessful after 81 times rebalance, :). So far I haven't found any issue. If I make a patch to move this parameter to be tunable in the proc filesystem on nommu case, is it acceptable? Thanks, -Aubrey - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Aubrey wrote: On 11/29/06, Nick Piggin <[EMAIL PROTECTED]> wrote: That was the order-9 allocation failure. Which is not going to be solved properly by just dropping caches. But Sonic apparently saw failures with 4K allocations, where the caches weren't getting shrunk properly. This would be more interesting because it would indicate a real problem with the kernel. I have done several test cases. when cat /proc/meminfo show MemFree < 8192KB, 1) malloc(1024 * 4), 256 times = 8MB, allocation successful. 2) malloc(1024 * 16), 64 times = 8MB, allocation successful. 3) malloc(1024 * 64), 16 times = 8MB, allocation successful. 4) malloc(1024 * 128), 8 times = 8MB, allocation failed. 5) malloc(1024 * 256), 4 times = 8MB, allocation failed. From those results, we know, when allocation <=64K, cache can be shrunk properly. That means the malloc size of an application on nommu should be <=64KB. That's exactly our problem. Some video programmes need a big block which has contiguous physical address. But yes, as you said, we must keep malloc not to alloc a big block to make the current kernel working robust on nommu. So, my question is, Can we improve this issue? why malloc(64K) is ok but malloc(128K) not? Is there any existing parameters about this issue? why not kernel attempt to shrunk cache no matter how big memory allocation is requested? Any thoughts? The pattern you are seeing here is probably due to the page allocator always retrying process context allocations which are <= order 3 (64K with 4K pages). You might be able to increase this limit a bit for your system, but it could easily cause problems. Especially fragmentation on nommu systems where the anonymous memory cannot be paged out. -- SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
On 11/29/06, Nick Piggin <[EMAIL PROTECTED]> wrote: That was the order-9 allocation failure. Which is not going to be solved properly by just dropping caches. But Sonic apparently saw failures with 4K allocations, where the caches weren't getting shrunk properly. This would be more interesting because it would indicate a real problem with the kernel. I have done several test cases. when cat /proc/meminfo show MemFree < 8192KB, 1) malloc(1024 * 4), 256 times = 8MB, allocation successful. 2) malloc(1024 * 16), 64 times = 8MB, allocation successful. 3) malloc(1024 * 64), 16 times = 8MB, allocation successful. 4) malloc(1024 * 128), 8 times = 8MB, allocation failed. 5) malloc(1024 * 256), 4 times = 8MB, allocation failed. From those results, we know, when allocation <=64K, cache can be shrunk properly. That means the malloc size of an application on nommu should be <=64KB. That's exactly our problem. Some video programmes need a big block which has contiguous physical address. But yes, as you said, we must keep malloc not to alloc a big block to make the current kernel working robust on nommu. So, my question is, Can we improve this issue? why malloc(64K) is ok but malloc(128K) not? Is there any existing parameters about this issue? why not kernel attempt to shrunk cache no matter how big memory allocation is requested? Any thoughts? Thanks, -Aubrey - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
On 11/29/06, Nick Piggin [EMAIL PROTECTED] wrote: That was the order-9 allocation failure. Which is not going to be solved properly by just dropping caches. But Sonic apparently saw failures with 4K allocations, where the caches weren't getting shrunk properly. This would be more interesting because it would indicate a real problem with the kernel. I have done several test cases. when cat /proc/meminfo show MemFree 8192KB, 1) malloc(1024 * 4), 256 times = 8MB, allocation successful. 2) malloc(1024 * 16), 64 times = 8MB, allocation successful. 3) malloc(1024 * 64), 16 times = 8MB, allocation successful. 4) malloc(1024 * 128), 8 times = 8MB, allocation failed. 5) malloc(1024 * 256), 4 times = 8MB, allocation failed. From those results, we know, when allocation =64K, cache can be shrunk properly. That means the malloc size of an application on nommu should be =64KB. That's exactly our problem. Some video programmes need a big block which has contiguous physical address. But yes, as you said, we must keep malloc not to alloc a big block to make the current kernel working robust on nommu. So, my question is, Can we improve this issue? why malloc(64K) is ok but malloc(128K) not? Is there any existing parameters about this issue? why not kernel attempt to shrunk cache no matter how big memory allocation is requested? Any thoughts? Thanks, -Aubrey - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Aubrey wrote: On 11/29/06, Nick Piggin [EMAIL PROTECTED] wrote: That was the order-9 allocation failure. Which is not going to be solved properly by just dropping caches. But Sonic apparently saw failures with 4K allocations, where the caches weren't getting shrunk properly. This would be more interesting because it would indicate a real problem with the kernel. I have done several test cases. when cat /proc/meminfo show MemFree 8192KB, 1) malloc(1024 * 4), 256 times = 8MB, allocation successful. 2) malloc(1024 * 16), 64 times = 8MB, allocation successful. 3) malloc(1024 * 64), 16 times = 8MB, allocation successful. 4) malloc(1024 * 128), 8 times = 8MB, allocation failed. 5) malloc(1024 * 256), 4 times = 8MB, allocation failed. From those results, we know, when allocation =64K, cache can be shrunk properly. That means the malloc size of an application on nommu should be =64KB. That's exactly our problem. Some video programmes need a big block which has contiguous physical address. But yes, as you said, we must keep malloc not to alloc a big block to make the current kernel working robust on nommu. So, my question is, Can we improve this issue? why malloc(64K) is ok but malloc(128K) not? Is there any existing parameters about this issue? why not kernel attempt to shrunk cache no matter how big memory allocation is requested? Any thoughts? The pattern you are seeing here is probably due to the page allocator always retrying process context allocations which are = order 3 (64K with 4K pages). You might be able to increase this limit a bit for your system, but it could easily cause problems. Especially fragmentation on nommu systems where the anonymous memory cannot be paged out. -- SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Aubrey wrote: On 11/29/06, Sonic Zhang <[EMAIL PROTECTED]> wrote: Forward to the mailing list. > On 11/27/06, Nick Piggin <[EMAIL PROTECTED]> wrote: >> I haven't actually written any nommu userspace code, but it is obvious >> that you must try to keep malloc to <= PAGE_SIZE (although order 2 and >> even 3 allocations seem to be reasonable, from process context)... Then >> you would use something a bit more advanced than a linear array to store >> data (a pagetable-like radix tree would be a nice, easy idea). >> > > But, even we split the 8M memory into 2048 x 4k blocks, we still face > this failure. The key problem is that available memory is small than > 2048 x 4k, while there are still a lot of VFS cache. The VFS cache can > be freed, but kernel allocation function ignores it. See the new test > application. Which kernel allocation function? If you can provide more details I'd like to get to the bottom of this. I posted it here, I think you missed it. So forwarded it to you. That was the order-9 allocation failure. Which is not going to be solved properly by just dropping caches. But Sonic apparently saw failures with 4K allocations, where the caches weren't getting shrunk properly. This would be more interesting because it would indicate a real problem with the kernel. Also, do you happen to know of a reasonable toolchain + emulator setup that I could test the nommu kernel with? A project named skyeye. http://www.skyeye.org/index.shtml Thanks, I'll give that one a try. Nick -- SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
On 11/29/06, Sonic Zhang <[EMAIL PROTECTED]> wrote: Forward to the mailing list. > On 11/27/06, Nick Piggin <[EMAIL PROTECTED]> wrote: >> I haven't actually written any nommu userspace code, but it is obvious >> that you must try to keep malloc to <= PAGE_SIZE (although order 2 and >> even 3 allocations seem to be reasonable, from process context)... Then >> you would use something a bit more advanced than a linear array to store >> data (a pagetable-like radix tree would be a nice, easy idea). >> > > But, even we split the 8M memory into 2048 x 4k blocks, we still face > this failure. The key problem is that available memory is small than > 2048 x 4k, while there are still a lot of VFS cache. The VFS cache can > be freed, but kernel allocation function ignores it. See the new test > application. Which kernel allocation function? If you can provide more details I'd like to get to the bottom of this. I posted it here, I think you missed it. So forwarded it to you. Because the anonymous memory allocation in mm/nommu.c is all allocated with GFP_KERNEL from process context, and in that case, the allocator should not fail but call into page reclaim which in turn will free VFS caches. > What's a better way to free the VFS cache in memory allocator? It should be freeing it for you, so I'm not quite sure what is going on. Can you send over the kernel messages you see when the allocation fails? I don't think so. The kernel doesn't attempt to free it. The log is included in the mail I forwarded to you. Also, do you happen to know of a reasonable toolchain + emulator setup that I could test the nommu kernel with? A project named skyeye. http://www.skyeye.org/index.shtml -Aubrey - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
On 11/29/06, Sonic Zhang [EMAIL PROTECTED] wrote: Forward to the mailing list. On 11/27/06, Nick Piggin [EMAIL PROTECTED] wrote: I haven't actually written any nommu userspace code, but it is obvious that you must try to keep malloc to = PAGE_SIZE (although order 2 and even 3 allocations seem to be reasonable, from process context)... Then you would use something a bit more advanced than a linear array to store data (a pagetable-like radix tree would be a nice, easy idea). But, even we split the 8M memory into 2048 x 4k blocks, we still face this failure. The key problem is that available memory is small than 2048 x 4k, while there are still a lot of VFS cache. The VFS cache can be freed, but kernel allocation function ignores it. See the new test application. Which kernel allocation function? If you can provide more details I'd like to get to the bottom of this. I posted it here, I think you missed it. So forwarded it to you. Because the anonymous memory allocation in mm/nommu.c is all allocated with GFP_KERNEL from process context, and in that case, the allocator should not fail but call into page reclaim which in turn will free VFS caches. What's a better way to free the VFS cache in memory allocator? It should be freeing it for you, so I'm not quite sure what is going on. Can you send over the kernel messages you see when the allocation fails? I don't think so. The kernel doesn't attempt to free it. The log is included in the mail I forwarded to you. Also, do you happen to know of a reasonable toolchain + emulator setup that I could test the nommu kernel with? A project named skyeye. http://www.skyeye.org/index.shtml -Aubrey - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Aubrey wrote: On 11/29/06, Sonic Zhang [EMAIL PROTECTED] wrote: Forward to the mailing list. On 11/27/06, Nick Piggin [EMAIL PROTECTED] wrote: I haven't actually written any nommu userspace code, but it is obvious that you must try to keep malloc to = PAGE_SIZE (although order 2 and even 3 allocations seem to be reasonable, from process context)... Then you would use something a bit more advanced than a linear array to store data (a pagetable-like radix tree would be a nice, easy idea). But, even we split the 8M memory into 2048 x 4k blocks, we still face this failure. The key problem is that available memory is small than 2048 x 4k, while there are still a lot of VFS cache. The VFS cache can be freed, but kernel allocation function ignores it. See the new test application. Which kernel allocation function? If you can provide more details I'd like to get to the bottom of this. I posted it here, I think you missed it. So forwarded it to you. That was the order-9 allocation failure. Which is not going to be solved properly by just dropping caches. But Sonic apparently saw failures with 4K allocations, where the caches weren't getting shrunk properly. This would be more interesting because it would indicate a real problem with the kernel. Also, do you happen to know of a reasonable toolchain + emulator setup that I could test the nommu kernel with? A project named skyeye. http://www.skyeye.org/index.shtml Thanks, I'll give that one a try. Nick -- SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Forward to the mailing list. Sonic Zhang wrote: On 11/27/06, Nick Piggin <[EMAIL PROTECTED]> wrote: I haven't actually written any nommu userspace code, but it is obvious that you must try to keep malloc to <= PAGE_SIZE (although order 2 and even 3 allocations seem to be reasonable, from process context)... Then you would use something a bit more advanced than a linear array to store data (a pagetable-like radix tree would be a nice, easy idea). But, even we split the 8M memory into 2048 x 4k blocks, we still face this failure. The key problem is that available memory is small than 2048 x 4k, while there are still a lot of VFS cache. The VFS cache can be freed, but kernel allocation function ignores it. See the new test application. Which kernel allocation function? If you can provide more details I'd like to get to the bottom of this. Because the anonymous memory allocation in mm/nommu.c is all allocated with GFP_KERNEL from process context, and in that case, the allocator should not fail but call into page reclaim which in turn will free VFS caches. What's a better way to free the VFS cache in memory allocator? It should be freeing it for you, so I'm not quite sure what is going on. Can you send over the kernel messages you see when the allocation fails? Also, do you happen to know of a reasonable toolchain + emulator setup that I could test the nommu kernel with? Thanks, Nick - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Forward to the mailing list. Sonic Zhang wrote: On 11/27/06, Nick Piggin [EMAIL PROTECTED] wrote: I haven't actually written any nommu userspace code, but it is obvious that you must try to keep malloc to = PAGE_SIZE (although order 2 and even 3 allocations seem to be reasonable, from process context)... Then you would use something a bit more advanced than a linear array to store data (a pagetable-like radix tree would be a nice, easy idea). But, even we split the 8M memory into 2048 x 4k blocks, we still face this failure. The key problem is that available memory is small than 2048 x 4k, while there are still a lot of VFS cache. The VFS cache can be freed, but kernel allocation function ignores it. See the new test application. Which kernel allocation function? If you can provide more details I'd like to get to the bottom of this. Because the anonymous memory allocation in mm/nommu.c is all allocated with GFP_KERNEL from process context, and in that case, the allocator should not fail but call into page reclaim which in turn will free VFS caches. What's a better way to free the VFS cache in memory allocator? It should be freeing it for you, so I'm not quite sure what is going on. Can you send over the kernel messages you see when the allocation fails? Also, do you happen to know of a reasonable toolchain + emulator setup that I could test the nommu kernel with? Thanks, Nick - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Aubrey wrote: On 11/22/06, Peter Zijlstra <[EMAIL PROTECTED]> wrote: The lack of a MMU on your system makes it very hard not to rely on higher order allocations, because even user-space allocs need to be physically contiguous. But please take that into consideration when writing software. Well, the test application just use an exaggerated way to replicate the issue. Actually, In the real work, the application such as mplayer, asterisk, etc will run into the above problem when run them at the second time. I think I have no reason to modify those kind of applications. No that's wrong. And your patch is just a hack that happens to mask the issue in the case you tested, and it will probably blow up in production at some stage (consider the case where the VFS cache page is not freeable or that page is being used for something else). With the nommu kernel, you actually *do* have a huge reason to write special code: large anonymous memory allocations have to use higher order allocations! I haven't actually written any nommu userspace code, but it is obvious that you must try to keep malloc to <= PAGE_SIZE (although order 2 and even 3 allocations seem to be reasonable, from process context)... Then you would use something a bit more advanced than a linear array to store data (a pagetable-like radix tree would be a nice, easy idea). You are of course free to put that patch into your product's kernel (although I would advise against it, because it has a lot of deadlock issues)... but the reality is that if you want a robust system, you cannot just take a unix program and run it unmodified on a nommu kernel AFAIKS. -- SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
On 11/22/06, Peter Zijlstra <[EMAIL PROTECTED]> wrote: Yes it does that, but there is no guarantee that those 50MB have a single 1M contiguous region amongst them. right ... the testcase posted is more to quickly illustrate the problem ... the requested size doesnt really matter, what does matter is that we cant seem to reclaim memory from the VFS cache in scenarios where the VFS cache is eating a ton of memory and we need some more another scenario is where an application is constantly reading data from a cd, re-encoding it to mp3, and then writing it to disk. the VFS cache here quickly eats up the available memory. -mike - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
On 11/22/06, Peter Zijlstra [EMAIL PROTECTED] wrote: Yes it does that, but there is no guarantee that those 50MB have a single 1M contiguous region amongst them. right ... the testcase posted is more to quickly illustrate the problem ... the requested size doesnt really matter, what does matter is that we cant seem to reclaim memory from the VFS cache in scenarios where the VFS cache is eating a ton of memory and we need some more another scenario is where an application is constantly reading data from a cd, re-encoding it to mp3, and then writing it to disk. the VFS cache here quickly eats up the available memory. -mike - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: The VFS cache is not freed when there is not enough free memory to allocate
Aubrey wrote: On 11/22/06, Peter Zijlstra [EMAIL PROTECTED] wrote: The lack of a MMU on your system makes it very hard not to rely on higher order allocations, because even user-space allocs need to be physically contiguous. But please take that into consideration when writing software. Well, the test application just use an exaggerated way to replicate the issue. Actually, In the real work, the application such as mplayer, asterisk, etc will run into the above problem when run them at the second time. I think I have no reason to modify those kind of applications. No that's wrong. And your patch is just a hack that happens to mask the issue in the case you tested, and it will probably blow up in production at some stage (consider the case where the VFS cache page is not freeable or that page is being used for something else). With the nommu kernel, you actually *do* have a huge reason to write special code: large anonymous memory allocations have to use higher order allocations! I haven't actually written any nommu userspace code, but it is obvious that you must try to keep malloc to = PAGE_SIZE (although order 2 and even 3 allocations seem to be reasonable, from process context)... Then you would use something a bit more advanced than a linear array to store data (a pagetable-like radix tree would be a nice, easy idea). You are of course free to put that patch into your product's kernel (although I would advise against it, because it has a lot of deadlock issues)... but the reality is that if you want a robust system, you cannot just take a unix program and run it unmodified on a nommu kernel AFAIKS. -- SUSE Labs, Novell Inc. Send instant messages to your online friends http://au.messenger.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
