Re: panic: kmem_map too small at heavy packet traffic
I have just pilled a PR. The negative written value is directly malloc's size parameter (in fact after some page size alignment enlargements operation). This parameter has been defined as unsigned long but printing with %ld as signed long. So if the size is very very big (more than 2^63 at amd64), the signed printing can remark the first bit of size as sign bit then write the - sign. But I think the size can not be so big, for this reason you are right, there must be a problem (the size parameter can come as negative or the enlargement functions can destroy the size parameter). On Sat, Jul 27, 2013 at 12:21 AM, Adrian Chadd adr...@freebsd.org wrote: Hi Have you filed a PR? This should get fixed. Also, being -ve is a problem. Is the value really negative? Is it wrapping badly? -adrian On 25 July 2013 07:57, Tugrul Erdogan h.tugrul.erdo...@gmail.com wrote: howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: panic: kmem_map too small at heavy packet traffic
Cool, what's the PR number? It sounds like something is odd. Email -current with the PR details (number, how you reproduce it, etc) and let's see if we can get one of the VM/UMA gurus to look into it. Thanks, -adrian On 27 July 2013 00:26, Tugrul Erdogan h.tugrul.erdo...@gmail.com wrote: I have just pilled a PR. The negative written value is directly malloc's size parameter (in fact after some page size alignment enlargements operation). This parameter has been defined as unsigned long but printing with %ld as signed long. So if the size is very very big (more than 2^63 at amd64), the signed printing can remark the first bit of size as sign bit then write the - sign. But I think the size can not be so big, for this reason you are right, there must be a problem (the size parameter can come as negative or the enlargement functions can destroy the size parameter). On Sat, Jul 27, 2013 at 12:21 AM, Adrian Chadd adr...@freebsd.org wrote: Hi Have you filed a PR? This should get fixed. Also, being -ve is a problem. Is the value really negative? Is it wrapping badly? -adrian On 25 July 2013 07:57, Tugrul Erdogan h.tugrul.erdo...@gmail.com wrote: howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: panic: kmem_map too small at heavy packet traffic
Specifically, I am taking this panic when doing ip spoof attack while syn-proxy activated. The output of system arguments below: kern.malloc_count: 315 vm.md_malloc_wait: 0 vfs.bufmallocspace: 0 vfs.maxmallocbufspace: 86269952 vm.kmem_size: 16686845952 vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size_scale: 1 vm.kmem_map_size: 543973376 vm.kmem_map_free: 15974895616 kern.maxvnodes: 350097 kern.minvnodes: 87524 vfs.numvnodes: 112329 vfs.wantfreevnodes: 87524 vfs.freevnodes: 87502 [root@ ~]# pfctl -si No ALTQ support in kernel ALTQ related functions disabled Status: Enabled for 0 days 00:17:39 Debug: Urgent State Table Total Rate current entries 5142886 searches2698214125478.9/s inserts 2905505327436.3/s removals2421865422869.4/s Counters match 2490130523514.0/s bad-offset 00.0/s fragment 00.0/s short 00.0/s normalize 00.0/s memory 00.0/s bad-timestamp 00.0/s congestion 00.0/s ip-option 180.0/s proto-cksum00.0/s state-mismatch 00.0/s state-insert 00.0/s state-limit00.0/s src-limit 00.0/s synproxy2937843927741.7/s [root@~]# panic: kmem_malloc(-1 814 425 600): kmem_map too small: 543 956 992 to tal allocated cpuid = 8 Uptime: 1d18h2m14s (ada0:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada0:ahcich1:0:0:0): CAM status: CCB request is in progress (ada0:ahcich1:0:0:0): Error 5, Retries exhausted (ada0:ahcich1:0:0:0): Synchronize cache failed (ada1:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada1:ahcich2:0:0:0): CAM status: CCB request is in progress (ada1:ahcich2:0:0:0): Error 5, Retries exhausted (ada1:ahcich2:0:0:0): Synchronize cache failed Dumping 8243 out of 16357 MB:..1%..11% On Thu, Jul 25, 2013 at 5:57 PM, Tugrul Erdogan h.tugrul.erdo...@gmail.comwrote: howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: panic: kmem_map too small at heavy packet traffic
I've been under the impression that synproxy was broken for quite some time, but I know there has been a lot of work on pf in HEAD so I can't be sure where it might stand there. Can anyone confirm/deny this? And not to discourage you, but the pf documentation does say Routine use of this option is not recommended, however, as it breaks expected TCP protocol behavior when the server can't process the request... However, panics are never good and hopefully someone can help you figure it out. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: panic: kmem_map too small at heavy packet traffic
Hi Have you filed a PR? This should get fixed. Also, being -ve is a problem. Is the value really negative? Is it wrapping badly? -adrian On 25 July 2013 07:57, Tugrul Erdogan h.tugrul.erdo...@gmail.com wrote: howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
panic: kmem_map too small at heavy packet traffic
howdy all, At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB ram. I am taking panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated message with configuration below: [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size: 16686845952 vm.kmem_size_scale: 1 [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem hw.physmem: 17151787008 hw.usermem: 8282652672 hw.realmem: 18253611008 [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages hw.pagesize: 4096 hw.pagesizes: 4096 2097152 0 hw.availpages: 4187448 When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 and after the panic at the core dump file the major vmstat difference is: temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 When I explore the source code of kernel (at vm_kern.c and vm_map.c), I see that the panic can occur with the cases at below: * negative malloc size parameter * longer than free buffer respect to kmem_map min_offset and max_offset values * try to allocate when the root entry of map is the rightmost entry of map * try to allocate bigger than map's max_free value I think the panic occurs at mbuf creation process when calling malloc() as a result of couldn't be able to allocate memory; but I don't understand why one of this panic case activating? The memory is almost empty but the device is saying kmem_map small when using about 0.5GB memory purely. How can i solve this panic problem? Thank you all for your time. -- Best Wishes, Tugrul Erdogan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
