Had another go when these fat httpd are running all gcore of a running httpd - pstack
feb4b6a4 __systemcall (ffbf7318, 4e, 1, 11, 1525e2d8, 1) + 34 fed6073c sendfilev64 (11, 1525e2d8, 1, ffbf7384, 0, 11000) + 1c fee9aeac apr_socket_sendfile (298518, 0, ffbf7384, 2000, ffbf948c, 1525e2d8) + 1d8 0003dffc ap_core_output_filter (1, 1cc, ffbf94a4, 35b56, 0, ffbf948c) + 110c 0005071c ap_http_header_filter (5e3768, 5e4ea8, 5e2998, 5e5048, 0, 0) + 848 00038484 default_handler (5e2998, 0, 5e4ea8, 64ad8, 2987b0, 0) + 468 0003e990 ap_run_handler (5e2998, 9, 27d1f8, 7f600, 27d2b0, 8) + 3c 0003efa8 ap_invoke_handler (5e2998, 27d330, 2310a0, 0, ffdcef60, 0) + e8 0004e1f4 ap_process_request (0, 7d000, 5e2998, 0, 0, 5e2998) + 54 0004b080 ap_process_http_connection (2987b0, 5e2998, c0bc0, 7d000, 7d134, 80000000) + 6c 00046a84 ap_run_process_connection (2987b0, 298518, 27ceb8, 7fe1c, 24d678, 0) + 3c 000529d8 child_main (7d000, 2987b0, 68adc, 7fe68, 7d334, 1) + 528 00052c98 make_child (52000, 4, 0, 0, 5, 7d000) + 13c 00052f3c perform_idle_server_maintenance (7d000, 6, 4, 7d324, 802b4, 1) + 1d0 00053748 ap_mpm_run (7d000, 0, 0, 4, 7d33c, ffbf9ca4) + 784 00027a64 main (c11e8, ffbffd90, 7d1b4, 7c120, 618f4, 7c110) + 9f4 00026b50 _start (0, 0, 0, 0, 0, 0) + 108 feb49984 __pollsys (ffbf7318, 1, ffbf72b0, 0, 1, 493e0) + 8 feaf5bec poll (ffbf7318, 1, 493e0, 10624dd3, 0, 0) + 7c feea1b08 apr_wait_for_io_or_timeout (0, 493e0, 0, 493e0, 0, 18) + dc fee9ae74 apr_socket_sendfile (298518, 0, ffbf94a4, 2000, ffbf948c, f827be0) + 1a0 0003dffc ap_core_output_filter (1, 90c, ffbf94a4, 35c31, 0, ffbf948c) + 110c 0005071c ap_http_header_filter (5e5ba8, 5e9728, 5e4dd8, 5e98c8, 0, 0) + 848 00038484 default_handler (5e4dd8, 0, 5e9728, 64ad8, 2987b0, 0) + 468 0003e990 ap_run_handler (5e4dd8, 9, 27d1f8, 7f600, 27d2b0, 8) + 3c 0003efa8 ap_invoke_handler (5e4dd8, 27d330, 2310a0, 0, ffdcef60, 0) + e8 0004e1f4 ap_process_request (0, 7d000, 5e4dd8, 0, 0, 5e4dd8) + 54 0004b080 ap_process_http_connection (2987b0, 5e4dd8, 0, 7d000, 7d134, 80000000) + 6c 00046a84 ap_run_process_connection (2987b0, 298518, 27ceb8, 7fe1c, 24d678, 0) + 3c 000529d8 child_main (7d000, 2987b0, 68adc, 7fe68, 7d334, 1) + 528 00052c98 make_child (52000, 6, 0, 0, 7, 7d000) + 13c 00052f3c perform_idle_server_maintenance (7d000, 8, 4, 7d324, 802b4, 1) + 1d0 00053748 ap_mpm_run (7d000, 0, 0, 2, 7d33c, ffbf9ca4) + 784 00027a64 main (c11e8, ffbffd90, 7d1b4, 7c120, 618f4, 7c110) + 9f4 00026b50 _start (0, 0, 0, 0, 0, 0) + 108 feb4b6a4 __systemcall (ffbf7318, 4e, 1, 11, 289e6b30, 1) + 34 fed6073c sendfilev64 (11, 289e6b30, 1, ffbf7384, 0, 11000) + 1c fee9aeac apr_socket_sendfile (298518, 0, ffbf7384, 2000, ffbf948c, 289e6b30) + 1d8 0003dffc ap_core_output_filter (1, 90c, ffbf94a4, 35c31, 0, ffbf948c) + 110c 0005071c ap_http_header_filter (5e5ba8, 5e9728, 5e4dd8, 5e98c8, 0, 0) + 848 00038484 default_handler (5e4dd8, 0, 5e9728, 64ad8, 2987b0, 0) + 468 0003e990 ap_run_handler (5e4dd8, 9, 27d1f8, 7f600, 27d2b0, 8) + 3c 0003efa8 ap_invoke_handler (5e4dd8, 27d330, 2310a0, 0, ffdcef60, 0) + e8 0004e1f4 ap_process_request (0, 7d000, 5e4dd8, 0, 0, 5e4dd8) + 54 0004b080 ap_process_http_connection (2987b0, 5e4dd8, 0, 7d000, 7d134, 80000000) + 6c 00046a84 ap_run_process_connection (2987b0, 298518, 27ceb8, 7fe1c, 24d678, 0) + 3c 000529d8 child_main (7d000, 2987b0, 68adc, 7fe68, 7d334, 1) + 528 00052c98 make_child (52000, 6, 0, 0, 7, 7d000) + 13c 00052f3c perform_idle_server_maintenance (7d000, 8, 4, 7d324, 802b4, 1) + 1d0 00053748 ap_mpm_run (7d000, 0, 0, 2, 7d33c, ffbf9ca4) + 784 00027a64 main (c11e8, ffbffd90, 7d1b4, 7c120, 618f4, 7c110) + 9f4 00026b50 _start (0, 0, 0, 0, 0, 0) + 108 Sriram Natarajan wrote: > This trace isn't very useful. You need to take the gcore image of > the fat process. This one does not look like that one. Are you running > with PHP ? If yes, what is the memory limit set to PHP runtime ? > > jpd wrote: >> Sriram Natarajan wrote: >> >>> Can you kindly send us the pstack output from the core image. >>> >>> pstack <core file> >>> >> >> feb4912c accept (5, 2986a0, 29868c, 1) >> 00054bf8 unixd_accept (ffbf9a48, c1860, 2984e0, 0, ffbf9a4c, 0) + 10 >> 000528f0 ???????? (7d000, 7d000, 68adc, 7fe68, 7d334, 1) >> 00052c98 ???????? (52000, 0, 0, 0, 1, 7d000) >> 00052f3c ???????? (7d000, 7, 4, 7d324, 802b4, 1) >> 00053748 ap_mpm_run (7d000, 0, 0, 0, 7d33c, ffbf9ca4) + 784 >> 00027a64 main (c11e8, ffbffd90, 7d1b4, 7c120, 618f4, 7c110) + 9f4 >> 00026b50 _start (0, 0, 0, 0, 0, 0) + 108 >> >> >> >>> or mdb <core file> followed by $G and $C . >>> >> >> ffbf9918 libc.so.1`_so_accept+4(ffbf99dc, c1880, 2984e0, 298518, >> feeb54f0, 2) >> ffbf9978 unixd_accept+0x10(ffbf9a48, c1860, 2984e0, 0, ffbf9a4c, 0) >> ffbf99e0 0x528f0(7d000, 7d000, 68adc, 7fe68, 7d334, 1) >> ffbf9a78 0x52c98(52000, 0, 0, 0, 1, 7d000) >> ffbf9ad8 0x52f3c(7d000, 7, 4, 7d324, 802b4, 1) >> ffbf9bd8 ap_mpm_run+0x784(7d000, 0, 0, 0, 7d33c, ffbf9ca4) >> ffbf9ca8 main+0x9f4(c11e8, ffbffd90, 7d1b4, 7c120, 618f4, 7c110) >> ffbffda8 _start+0x108(0, 0, 0, 0, 0, 0) >> >> >> >>> - Sriram >>> >>> John-Paul Drawneek wrote: >>> >>>> Ok on my web boxes recently a really fat httpd instance keeps appearing. >>>> >>>> biggest i have seen is 1G swap 500M real - which is huge compared to the >>>> rest. >>>> >>>> I got some core of these - so any idea how to did out where the memory is >>>> going? >>>> >>>> And then to work out whats setting it off. >>>> >>>> Really need some help on this :( >>>> This message posted from opensolaris.org >>>> >>>> _______________________________________________ >>>> >>>> >>>> webstack-discuss mailing list >>>> webstack-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/webstack-discuss >>>> >>>> >> >> _______________________________________________ >> >> >> webstack-discuss mailing list >> webstack-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/webstack-discuss >>
