Re: [osv-dev] Re: OOM query

2020-03-22 Thread Rick Payne
On Sun, 2020-03-22 at 22:08 -0700, Waldek Kozaczuk wrote: > > > On Monday, March 23, 2020 at 12:36:52 AM UTC-4, rickp wrote: > > Looks to me like its trying to allocate 40MB but the available > > memory > > is 10GB, surely? 10933128KB is 10,933MB > > > > I misread the number - forgot about 1K

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Waldek Kozaczuk
Based on what I can tell the available remaining memory seems to be heavily fragmented (I have no idea why). If it is it would help to implement https://github.com/cloudius-systems/osv/issues/854 which I will give it a try. Hopefully, I can have a patch within a day. On Monday, March 23, 2020 a

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Waldek Kozaczuk
On Monday, March 23, 2020 at 12:36:52 AM UTC-4, rickp wrote: > > > Looks to me like its trying to allocate 40MB but the available memory > is 10GB, surely? 10933128KB is 10,933MB > > I misread the number - forgot about 1K. Any chance you could run the app outside of production with memory trac

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Rick Payne
Looks to me like its trying to allocate 40MB but the available memory is 10GB, surely? 10933128KB is 10,933MB Its probably erlang garbage collection, but it seems very strange to me as sometimes it happens very quickly after we start the system. There's no way we're allocate more memory than the

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Waldek Kozaczuk
Do you know if the application uses "huge" (>=2MB) malloc? I found this relevant discussion - https://groups.google.com/d/msg/osv-dev/uLoxcjU164k/WWl6dMzsCAAJ - about how OSv currently uses contiguous physical memory areas to back these kinds of requests. There is an open issue about that -

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Waldek Kozaczuk
Either way, it looks like the app is trying to allocate almost 40MB and the remaining memory is ~ 10MB. So, in this case, there is no surprise, is there? In the previous case, you reported there were two threads each requesting ~5MB and there was plenty of memory. If I understand the code corre

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Waldek Kozaczuk
It does for me. Are you using the latest version of this script from master? I think there were couple of things I changed there recently as part of wide upgrade to Python 3. On Sun, Mar 22, 2020 at 20:27 Rick Payne wrote: > > Does that command work for you? For me I get: > > (gdb) osv heap > Py

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Rick Payne
Does that command work for you? For me I get: (gdb) osv heap Python Exception %x format: an integer is required, not gdb.Value: Error occurred in Python command: %x format: an integer is required, not gdb.Value Rick On Sun, 2020-03-22 at 19:37 -0400, Waldek Kozaczuk wrote: > Can you run ‘os

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Waldek Kozaczuk
Can you run ‘osv heap’ from gdb at this point? On Sun, Mar 22, 2020 at 19:31 Rick Payne wrote: > > Ok, so i applied the patch, and the printf and this is what I see: > > page_range_allocator: no ranges found for size 39849984 and exact > order: 14 > Waiter: 1_scheduler, bytes: 39849984 > Out of

Re: [osv-dev] Re: OOM query

2020-03-22 Thread Rick Payne
Ok, so i applied the patch, and the printf and this is what I see: page_range_allocator: no ranges found for size 39849984 and exact order: 14 Waiter: 1_scheduler, bytes: 39849984 Out of memory: could not reclaim any further. Current memory: 10933128 Kb, target: 0 Kb Rick On Sat, 2020-03-21 at

[osv-dev] [PATCH 2/2] tests: move nfs testing out to a separate module

2020-03-22 Thread Waldemar Kozaczuk
This patch moves nfs testing logic from ./scripts/test.py into ./modules/nfs-tests/test.py. It also refreshes nfs-tests to pull and build UNFS3 (NFS in user space) from a new repo. Eventually it adds new 'check' target that builds the test image (nfs,nfs-tests) and runs test.py against it. Signed

[osv-dev] [PATCH 1/2] fs: move nfs support out of kernel in lieu of a separate pluggable module

2020-03-22 Thread Waldemar Kozaczuk
This patch removes external/fs/libnfs module and makes nfs support pluggable by moving it into a module (shared library) that can be added to the image instead of being compiled into the kernel using nfs=true build option. The nfs support can be added by adding nfs module to the image. More specif