how to recreate the file descriptor segfault? Re: cvs commit:httpd-2.0 STATUS
I just tried to debug this with the current CVS HEAD on Solaris, but I can't reproduce the crash. The test case that I tried was: set the httpd's ulimit on file descriptors to a small number and run ab to generate enough concurrent requests to exceed the fd limit. Is there a better test case that will trigger the segfault? Thanks, --Brian Jeff Trawick wrote: >[EMAIL PROTECTED] writes: > >>brianp 02/01/11 00:07:07 >> >> Modified:.STATUS >> Log: >> Updated STATUS to cover the worker segfault fixes >> >> > >> -* The worker MPM on Solaris segfaults when it runs out of file >> - descriptors. (This may affect other MPMs and/or platforms.) >> > >I can still readily hit this on current code (the same code that no >longer segfaults with graceful restart). > >[Fri Jan 11 07:26:37 2002] [error] (24)Too many open files: >apr_accept: (client socket) >[Fri Jan 11 07:26:37 2002] [error] [client 127.0.0.1] (24)Too many >open files: file permissions deny server access: /exp >ort/home/trawick/apacheinst/htdocs/index.html.en >[Fri Jan 11 07:26:37 2002] [error] [client 127.0.0.1] (24)Too many >open files: cannot access type map file: /export/home >/trawick/apacheinst/error/HTTP_FORBIDDEN.html.var >[Fri Jan 11 07:26:38 2002] [notice] child pid 25493 exit signal >Segmentation fault (11), possible coredump in /export/ho >me/trawick/apacheinst > >This is the same coredump I saw before: > >#0 0xff33a3cc in apr_wait_for_io_or_timeout (sock=0x738360, > for_read=1) at sendrecv.c:70 >70 FD_SET(sock->socketdes, &fdset); > >The socket has already been closed so trying to set bit -1 segfaults. >
Re: [PATCH] fix segfault in core input filter
Justin Erenkrantz wrote: > On Fri, Jan 11, 2002 at 03:14:58PM -0500, Jeff Trawick wrote: > >>>If the brigade in core_input_filter is empty, it means that the >>>socket (and its bucket) has disappeared on us. Most likely, >>>someone isn't paying attention to APR_EOF return values. >>> >>>We should return an error or at the very least APR_EOF if we >>>see an empty brigade there. But, it would be more useful to >>>find out which input filter isn't paying attention. -- justin >>> >>Does anybody have any concerns before I commit? >> > > Looks good to me. +1. -- justin > > does the proxy break? it usually does whenever the input filter changes ;(
Re: [PATCH] fix segfault in core input filter
On Fri, Jan 11, 2002 at 03:14:58PM -0500, Jeff Trawick wrote: > > If the brigade in core_input_filter is empty, it means that the > > socket (and its bucket) has disappeared on us. Most likely, > > someone isn't paying attention to APR_EOF return values. > > > > We should return an error or at the very least APR_EOF if we > > see an empty brigade there. But, it would be more useful to > > find out which input filter isn't paying attention. -- justin > > Does anybody have any concerns before I commit? Looks good to me. +1. -- justin
new httpd-2_0_28 build running on daedalus
...as of Fri Jan 11 15:05:33. The purpose of this build was to see if the load spikes happen with any build done on FreeBSD 4.5 because of our problems with more recent httpd code. So far I haven't seen the spikes. Also, my patch to log cpu per request is running. Greg
Re: FreeBSD load statistics.
Brian Pane wrote: > > Greg Ames wrote: > > >>On Wed, Jan 09, 2002 at 11:55:21PM -0800, Brian Pane wrote: > >> > >>>My hypothesis upon seeing these numbers is that the high run queue length > >>>is due to us hitting some bottleneck in the kernel on daedalus--and because > >>>post-2.0.28 releases waste less time in usr-space code between syscalls, > >>>they hit that kernel bottleneck a lot harder than 2.0.28 did. > >>> > > > >An interesting theory. But it doesn't explain the high numbers of CPU ticks > >I've recorded for trivial requests, nor does it explain the top Brian B sent us > >with two httpd's both using over 50% of the CPU for a while > > > > It might explain the high number of CPU ticks for simple requests if > a lot of processes are hitting, say, a spin lock at the same time. But > the two httpds using 50% of the CPU are definitely something different. OK, spin locks sound like a reasonable explanation. I'm surprised that they spin that long though. I have 2_0_28 with cpu logging up now on port 8092. I ran log replay against it, and saw one simple request that used 221 cpu ticks (1.7 sec). Everything else was 20 ticks or less. So either this happens all the time and is a red herring, or this build has got the problem too. I suspect it's the former, but will put it into production later to be sure. > We really need some data on what's happening in those processes on daedalus. Yessir. I'd love it if there were something like ktrace that wrote to a big array in memory (i.e. not to a file) that we could grab if the run queue exceeded, say, 180. Greg
Re: [PATCH] fix segfault in core input filter
Justin Erenkrantz <[EMAIL PROTECTED]> writes:
> On Fri, Jan 11, 2002 at 09:03:36AM -0500, Jeff Trawick wrote:
> > With current CVS code, prefork, APR_POOL_DEBUG, Linux, and
> > ElectricFence I readily hit a segfault on the APR_BRIGADE_NORMALIZE()
> > call in core input filter. It segfaulted because the brigade was
> > empty. With the following patch I can't seem to hit a segfault
> > anymore.
> >
> > Perhaps the brigade shouldn't be empty for some reason...
>
> If the brigade in core_input_filter is empty, it means that the
> socket (and its bucket) has disappeared on us. Most likely,
> someone isn't paying attention to APR_EOF return values.
>
> We should return an error or at the very least APR_EOF if we
> see an empty brigade there. But, it would be more useful to
> find out which input filter isn't paying attention. -- justin
Does anybody have any concerns before I commit?
(I should run with ElectricFence more often...)
Index: server/core.c
===
RCS file: /home/cvs/httpd-2.0/server/core.c,v
retrieving revision 1.129
diff -u -r1.129 core.c
--- server/core.c 11 Jan 2002 18:55:27 - 1.129
+++ server/core.c 11 Jan 2002 20:13:48 -
@@ -3035,6 +3035,10 @@
APR_BRIGADE_INSERT_TAIL(ctx->b, e);
net->in_ctx = ctx;
}
+else if (APR_BRIGADE_EMPTY(ctx->b)) {
+/* hit EOF on socket already */
+return APR_EOF;
+}
/* ### This is bad. */
APR_BRIGADE_NORMALIZE(ctx->b);
@@ -3114,9 +3118,6 @@
AP_DEBUG_ASSERT(*readbytes > 0);
-if (APR_BRIGADE_EMPTY(ctx->b))
-return APR_EOF;
-
e = APR_BRIGADE_FIRST(ctx->b);
rv = apr_bucket_read(e, &str, &len, mode);
--
Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: [PATCH] fix segfault in core input filter
"Bill Stoddard" <[EMAIL PROTECTED]> writes:
> > @@ -260,6 +261,12 @@
> > */
> > APR_BRIGADE_INSERT_HEAD(bb, e);
> > ap_pass_brigade(r->connection->output_filters, bb);
> > +if (rv != APR_SUCCESS) {
> > +/* We got an error earlier reading from the socket. Make sure
> > + * keepalive is off so that we don't try to read another request.
> > + */
> > +r->connection->keepalive = 0;
> > +}
> > }
> > }
> >
>
> Should we set r->connection->aborted here?
no, because that disables lingering close... we just need to know not
to bother trying to read more requests from the socket, and clearing
keepalive does that
but I'm working on a simpler patch now anyway which handles both of
the paths I found where APR_EOF was getting lost
--
Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: [PATCH] fix segfault in core input filter
Jeff Trawick <[EMAIL PROTECTED]> writes: > The patch below gets rid of a segfault while processing every HTTP/1.1 > request (w/ ElectricFence and APR_POOL_DEBUG) but there is a much less > infrequent segfault remaining :( The infrequent segfault still remaining is from a scenario where ap_rgetline() gets APR_SUCCESS from a call to ap_get_brigade() which hit EOF on the socket. 1) pid 14795 status 3 2) 14795 getting brigade in ap_rgetline() 3) hit socket error 20514 in 14795, returning immortal 4) 14795 getting brigade returned 0 5) 14795 getting brigade in ap_rgetline() 6) [Fri Jan 11 14:41:47 2002] [notice] child pid 14795 exit signal Seg... 1: entering BUSY_READ state 2: ap_rgetline() about to call ap_get_brigade() 3: socket_read() hits EOF on socket, returns immortal "" bucket 4: ap_get_brigade() returns APR_SUCCESS to ap_rgetline() (it has a valid line) 5: ap_rgetline() about to call ap_get_brigade() 6: segfault in core input because ctx->b is empty It would seem that the simplest patch to solve the last two problems would be to check the brigade near the start of core-input and return APR_EOF if it is empty. (Justin suggested that some hours ago :) ). -- Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site: http://www.geocities.com/SiliconValley/Park/9289/ Born in Roswell... married an alien...
Re: [PATCH] fix segfault in core input filter
> @@ -260,6 +261,12 @@
> */
> APR_BRIGADE_INSERT_HEAD(bb, e);
> ap_pass_brigade(r->connection->output_filters, bb);
> +if (rv != APR_SUCCESS) {
> +/* We got an error earlier reading from the socket. Make sure
> + * keepalive is off so that we don't try to read another request.
> + */
> +r->connection->keepalive = 0;
> +}
> }
> }
>
Should we set r->connection->aborted here?
Bill
Re: Solaris 2.5.1 builds?
On Fri, Jan 11, 2002 at 06:47:03PM +0100, Mladen Turk wrote: [snip] > using gcc3.0.2 > For a start just tried to build the apr. > > bash-2.05$ ./configure --prefix=/whatever \ > --build=sparc-sun-solaris2.251 > --with-gnu-ld \ > --disable-ipv6 > ... > bash-2.05$ make > ... > bash-2.05$ make test > ... > bash-2.05$ setenv LD_LIBRARY_PATH /usr/local/lib > bash-2.05$ ./testfile Does libgcc_s.so* live in /usr/local/lib? I don't see your LDFLAGS line in your posting, but I'm assuming you set it. [snip] > All tests passed OK > Segmentation Fault (core dumped) > bash-2.05$ > > > > Well core gets dumped for all test, but interesting almost all the test are > passed except > the testsock which complains that libgcc_s.so.1 open failed. The core dumps are interesting, can you post a stack trace? As for testsock, it's simply not finding libgcc_s.so.1. Try running ldd .libs/testsock (or ldd .libs/lt-testsock if you have it). -aaron
Re: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?
On Fri, Jan 11, 2002 at 01:32:35PM -0400, Marc G. Fournier wrote: > On Fri, 11 Jan 2002, Aaron Bannert wrote: > > > On Fri, Jan 11, 2002 at 10:27:18AM -0400, Marc G. Fournier wrote: > > ... > > > [Fri Jan 11 14:20:19 2002] [crit] (78)Function not implemented: Fatal error: >could not open(create) scoreboard > > ... > > > > > Even if I comment out the 'ScoreBoard' line in httpd.conf, it > > > gives the 'Function not implemented' error ... > > > > > > Known problem? If not, suggestions on how to debug and provide > > > something just a wee bit more useful? I'm not finding any core files > > > laying about, so nothing I can gdb ... > > ... > > > > send in the output of `grep SHM srclib/apr/include/apr.h` so we can find > > out what shared memory subsystems are available on your platform. > > jail# grep SHM srclib/apr/include/apr.h > #define APR_HAVE_SHMEM_MMAP_TMP 1 > #define APR_HAVE_SHMEM_MMAP_SHM 1 > #define APR_HAVE_SHMEM_MMAP_ZERO1 > #define APR_HAVE_SHMEM_SHMGET_ANON 1 > #define APR_HAVE_SHMEM_SHMGET 1 > #define APR_HAVE_SHMEM_MMAP_ANON1 > #define APR_HAVE_SHMEM_BEOS 0 > #define APR_USE_SHMEM_MMAP_TMP 0 > #define APR_USE_SHMEM_MMAP_SHM 0 > #define APR_USE_SHMEM_MMAP_ZERO0 > #define APR_USE_SHMEM_SHMGET_ANON 0 > #define APR_USE_SHMEM_SHMGET 1 > #define APR_USE_SHMEM_MMAP_ANON1 > #define APR_USE_SHMEM_BEOS 0 These are identical to what I have on 5.0-CURRENT, and I just verified that it still works for me. If you are getting "Not implemented on this platform" errors, then something funny is going on -- that only happens in the scoreboard if you don't have any anonymous shmem implementations, which according to your HAVE_SHMEM defines above is not true (you have every shmem type known to APR on your system). I'd say first try doing a full "make clean && make distclean && ./buildconf" to make sure nothing is stale. Let us know if you're still seeing this problem at that point. Like Justin suggested, this could just be a problem with the jail. If the above doesn't work, you might try changing to these defines: #define APR_USE_SHMEM_SHMGET_ANON 1 #define APR_USE_SHMEM_MMAP_ANON0 So that you're using shmget() for the ANON shmem instead of mmap(). -aaron
Re: FreeBSD load statistics.
Aaron Bannert wrote: > Another thing that was brought up yesterday at dinner was the fact > that Daedalus underwent a system upgrade to 4.5. AIUI this also means > that things like libc and gcc were updated, and means that 2.0.28 and > 2.0.30 were compiled by different compilers. Perhaps we should recompile > 2.0.28 under the new environment and see if the load spikes are still > not happening. That would at least exclude the possibilitity that this > was introduced by the compiler and/or libraries as part of the upgrade. hmmm... 2.0.30-dev went live on Dec 21. On Dec 28 Brian B reported to the infrastructure list that he had upgraded daedalus to 4.5, and had been seeing load spikes for the last week. But I want to do another 2_0_28 build anyway to stick in my cpu per request logging patch, to see if the phenomenon where some trivial requests use a lot of cpu exists there as well. This will kill two birds with one stone. Greg
Re: Apache 2.0.28 memory growth issue.
On Fri, Jan 11, 2002 at 01:07:00PM -0500, Greg Ames wrote: > Aaron Bannert wrote: > > > Keep in mind that the multithreaded MPMs will by their very nature have > > higher memory requirements than their single-threaded counterparts. I > > regularly see memory usage in the range you've reported. Yesterday > > while running with 500 threads, I can report that httpd was using 23M > > (20M RSS). If you consider that 500 prefork processes would take on > > the order 500*N, where N is usually 3-5MB, then this is a *significant* > > decrease in memory requirements -- which was one of the major goals of > > the hybrid multiprocess/ multithreaded httpd model. > > Good point. I can see it while worker is getting pounded. > > hmmmn, but what if the traffic dries up completely? Should we expect memory > usage to shrink back down close to where it started? Nope. Because you can never give memory back to the system, whatever top shows for the SIZE is always a floor. The best you can do is reuse memory (as we do with our pool freelists, and as free() does in libc). The system may decide to steal unused pages from httpd, which shows up in the discrepancy between SIZE and RSS (or RES on solaris) on the top output, but as soon as those pages become active again those two columns should converge. -aaron
Re: FreeBSD load statistics.
Greg Ames wrote: >>On Wed, Jan 09, 2002 at 11:55:21PM -0800, Brian Pane wrote: >> >>>My hypothesis upon seeing these numbers is that the high run queue length >>>is due to us hitting some bottleneck in the kernel on daedalus--and because >>>post-2.0.28 releases waste less time in usr-space code between syscalls, >>>they hit that kernel bottleneck a lot harder than 2.0.28 did. >>> > >An interesting theory. But it doesn't explain the high numbers of CPU ticks >I've recorded for trivial requests, nor does it explain the top Brian B sent us >with two httpd's both using over 50% of the CPU for a while > It might explain the high number of CPU ticks for simple requests if a lot of processes are hitting, say, a spin lock at the same time. But the two httpds using 50% of the CPU are definitely something different. We really need some data on what's happening in those processes on daedalus. --Brian
Re: [PATCH] fix segfault in core input filter
Jeff Trawick <[EMAIL PROTECTED]> writes:
> I'm following what happens when we get EOF on socket...
>
> socket_read() returns APR_SUCCESS and immortal bucket with ""
>
> core-input doesn't error out since it got APR_SUCCESS
>
> core-input walks over the "" and deletes the immortal bucket
>
> core-input sees empty brigade and returns APR_EOF
>
> it gets returned up through net_time_filter to check_pipeline_flush
>
> check_pipeline_flush knows ap_get_brigade() failed but it lost the
> retcode and has no way to return the error anyway
>
> return to caller (ap_process_request)
>
> do ap_run_log_transaction
>
> back to ap_process_http_connection
>
> ap_process_http_connection calls ap_read_request again and we segfault
> since core-input's brigade was empty
>
> --/--
>
> if check_pipeline_flush cleared c->keepalive we wouldn't have tried to
> read another request
The patch below gets rid of a segfault while processing every HTTP/1.1
request (w/ ElectricFence and APR_POOL_DEBUG) but there is a much less
infrequent segfault remaining :(
Index: modules/http/http_request.c
===
RCS file: /home/cvs/httpd-2.0/modules/http/http_request.c,v
retrieving revision 1.122
diff -u -r1.122 http_request.c
--- modules/http/http_request.c 14 Dec 2001 03:29:13 - 1.122
+++ modules/http/http_request.c 11 Jan 2002 18:26:04 -
@@ -240,6 +240,7 @@
### need to send a FLUSH. */
apr_bucket_brigade *bb = apr_brigade_create(r->pool);
apr_off_t zero = 0;
+apr_status_t rv = APR_SUCCESS;
/* Flush the filter contents if:
*
@@ -249,7 +250,7 @@
/* ### shouldn't this read from the connection input filters? */
/* ### is zero correct? that means "read one line" */
if (!r->connection->keepalive ||
-ap_get_brigade(r->input_filters, bb, AP_MODE_PEEK, &zero) != APR_SUCCESS) {
+(rv = ap_get_brigade(r->input_filters, bb, AP_MODE_PEEK, &zero)) !=
+APR_SUCCESS) {
apr_bucket *e = apr_bucket_flush_create();
/* We just send directly to the connection based filters. At
@@ -260,6 +261,12 @@
*/
APR_BRIGADE_INSERT_HEAD(bb, e);
ap_pass_brigade(r->connection->output_filters, bb);
+if (rv != APR_SUCCESS) {
+/* We got an error earlier reading from the socket. Make sure
+ * keepalive is off so that we don't try to read another request.
+ */
+r->connection->keepalive = 0;
+}
}
}
--
Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: Apache 2.0.28 memory growth issue.
> Aaron Bannert wrote: > > > Keep in mind that the multithreaded MPMs will by their very nature have > > higher memory requirements than their single-threaded counterparts. I > > regularly see memory usage in the range you've reported. Yesterday > > while running with 500 threads, I can report that httpd was using 23M > > (20M RSS). If you consider that 500 prefork processes would take on > > the order 500*N, where N is usually 3-5MB, then this is a *significant* > > decrease in memory requirements -- which was one of the major goals of > > the hybrid multiprocess/ multithreaded httpd model. > > Good point. I can see it while worker is getting pounded. > > hmmmn, but what if the traffic dries up completely? Should we expect memory > usage to shrink back down close to where it started? > > Greg > Depends on the implementation details of the worker MPM. Specifically, do you preallocate a ptrans pool for each and every worker thread or allocate one only when the thread is dispatched to handle a request? Once a ptrans pool is allocated, do you free it when the thread goes back into the worker pool? Bill
Re: FreeBSD load statistics.
> On Wed, Jan 09, 2002 at 11:55:21PM -0800, Brian Pane wrote: > > My hypothesis upon seeing these numbers is that the high run queue length > > is due to us hitting some bottleneck in the kernel on daedalus--and because > > post-2.0.28 releases waste less time in usr-space code between syscalls, > > they hit that kernel bottleneck a lot harder than 2.0.28 did. An interesting theory. But it doesn't explain the high numbers of CPU ticks I've recorded for trivial requests, nor does it explain the top Brian B sent us with two httpd's both using over 50% of the CPU for a while Justin Erenkrantz wrote: > FWIW, Aaron was telling me that he remembers seeing high run-queue > lengths on machines with failing hard drives. OK, but daedalus uses the same hard drives under 2_0_28. Greg
Re: [PATCH] fix segfault in core input filter
Justin Erenkrantz <[EMAIL PROTECTED]> writes: > On Fri, Jan 11, 2002 at 09:03:36AM -0500, Jeff Trawick wrote: > > With current CVS code, prefork, APR_POOL_DEBUG, Linux, and > > ElectricFence I readily hit a segfault on the APR_BRIGADE_NORMALIZE() > > call in core input filter. It segfaulted because the brigade was > > empty. With the following patch I can't seem to hit a segfault > > anymore. > > > > Perhaps the brigade shouldn't be empty for some reason... > > If the brigade in core_input_filter is empty, it means that the > socket (and its bucket) has disappeared on us. Most likely, > someone isn't paying attention to APR_EOF return values. > > We should return an error or at the very least APR_EOF if we > see an empty brigade there. But, it would be more useful to > find out which input filter isn't paying attention. -- justin I'm following what happens when we get EOF on socket... socket_read() returns APR_SUCCESS and immortal bucket with "" core-input doesn't error out since it got APR_SUCCESS core-input walks over the "" and deletes the immortal bucket core-input sees empty brigade and returns APR_EOF it gets returned up through net_time_filter to check_pipeline_flush check_pipeline_flush knows ap_get_brigade() failed but it lost the retcode and has no way to return the error anyway return to caller (ap_process_request) do ap_run_log_transaction back to ap_process_http_connection ap_process_http_connection calls ap_read_request again and we segfault since core-input's brigade was empty --/-- if check_pipeline_flush cleared c->keepalive we wouldn't have tried to read another request any other ideas? -- Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site: http://www.geocities.com/SiliconValley/Park/9289/ Born in Roswell... married an alien...
Re: Apache 2.0.28 memory growth issue.
Aaron Bannert wrote: > Keep in mind that the multithreaded MPMs will by their very nature have > higher memory requirements than their single-threaded counterparts. I > regularly see memory usage in the range you've reported. Yesterday > while running with 500 threads, I can report that httpd was using 23M > (20M RSS). If you consider that 500 prefork processes would take on > the order 500*N, where N is usually 3-5MB, then this is a *significant* > decrease in memory requirements -- which was one of the major goals of > the hybrid multiprocess/ multithreaded httpd model. Good point. I can see it while worker is getting pounded. hmmmn, but what if the traffic dries up completely? Should we expect memory usage to shrink back down close to where it started? Greg
Re: Apache 2.0.28 memory growth issue.
Chris Williams wrote: > If I want to get the tag'd version of 2.0.30 from CVS, what is the > command? The only thing I saw on the developer page was how to get the > latest and greatest. cvs checkout -r APACHE_2_0_30 httpd-2.0 cd httpd-2.0/srclib cvs checkout -r APACHE_2_0_30 apr apr-util Greg
RE: Apache 2.0.28 memory growth issue.
Thanks for the response Ian. I have one newbie type of question. I have been using the beta's that apache has been posting which the latest was 2.0.28. If I want to get the tag'd version of 2.0.30 from CVS, what is the command? The only thing I saw on the developer page was how to get the latest and greatest. Thanks for your help Chris > -Original Message- > From: Ian Holsman [mailto:[EMAIL PROTECTED]] > Sent: Friday, January 11, 2002 11:24 AM > To: [EMAIL PROTECTED]; Chris Williams > Subject: Re: Apache 2.0.28 memory growth issue. > > > Chris Williams wrote: > > > I am running apache 2.0.28 on Solaris 8 with the worker MPM. I > am seeing > > memory growth when I use flood to hammer apache when just > serving up static > > pages. I have had flood running with 250 farmers constantly running the > > past hour and the growth of the httpd processes using the ps command has > > gone from 4200 to 15000 when using the RSS value from ps. Was > this growth > > normal in 2.0.28? I know the latest tagged version is 2.0.30 > and wonder if > > I should try and get it setup on my Solaris box(doesn't have libtool and > > other stuff needed to build from CVS). Is/was this a known > issue and will > > upgrading to a version later than 2.0.28 help? One other > question...what is > > the correct syntax for the CVS command to get a specific label > out of CVS > > sat I want to get 2.0.30 instead of the HEAD? > > Thanks! > > Chris > > > > > > Hi Chris. > I'm not sure about .28, but .29 (+aprtableget patch from 30) seems > pretty stable for us, and we haven't seen that memory growth you > describe. does the RSS size constantly grow from the start, or does > it sit still for a period of time and then start increasing. > also it could be a module you are running. >
Re: Solaris 2.5.1 builds?
- Original Message - From: "Justin Erenkrantz" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 11, 2002 2:09 AM Subject: Re: Solaris 2.5.1 builds? > That should fix it. If not, find libgcc_s.so.1 and add > -R/path/to/libgcc/dir to LDFLAGS. -- justin OK... did using gcc3.0.2 For a start just tried to build the apr. bash-2.05$ ./configure --prefix=/whatever \ --build=sparc-sun-solaris2.251 --with-gnu-ld \ --disable-ipv6 ... bash-2.05$ make ... bash-2.05$ make test ... bash-2.05$ setenv LD_LIBRARY_PATH /usr/local/lib bash-2.05$ ./testfile APR File Functions Test === Initializing APROK Creating the main pool we'll useOK Creating the second pool we'll use OK Testing file functions. Opening fileOK Checking filename test.fil Writing to the file OK Checking we wrote everythingOK Moving to the start of file OK This platform supports files_like_sockets, testing... Making file look like a socket OK Checking for incoming data OK End of files as sockets test. Reading from the file OK Checking what we read OK Adding user data to the fileOK Getting user data from the file OK Checking the data we gotOK Getting fileinfoOK Closing the fileOK Stat'ing file OK Getting groupname OK Comparing group ID'sOK (gid's for other match) Getting usernameOK Comparing users OK (uid's for root match) Deleting the file OK Making sure it's gone OK Testing Directory functions. Making directoryOK Creating a file in the new directoryOK Opening directory OK Reading directory OK Getting Information about the file... File name OK File type OK File size OK Done checking file information Rewind directoryOK Closing directory OK Removing file from directoryOK Removing directory OK Creating the file OK Closing the fileOK Removing the file OK Checking it's gone OK Testing file read functions. Buffered file tests OK Unbuffered file tests OK More buffered file testsOK More unbuffered file tests OK Even more buffered file tests OK Even more unbuffered file tests OK All read tests OK All tests passed OK Segmentation Fault (core dumped) bash-2.05$ Well core gets dumped for all test, but interesting almost all the test are passed except the testsock which complains that libgcc_s.so.1 open failed. Any clues? MT.
Re: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?
On Fri, 11 Jan 2002, Aaron Bannert wrote: > On Fri, Jan 11, 2002 at 10:27:18AM -0400, Marc G. Fournier wrote: > ... > > [Fri Jan 11 14:20:19 2002] [crit] (78)Function not implemented: Fatal error: could >not open(create) scoreboard > ... > > > Even if I comment out the 'ScoreBoard' line in httpd.conf, it > > gives the 'Function not implemented' error ... > > > > Known problem? If not, suggestions on how to debug and provide > > something just a wee bit more useful? I'm not finding any core files > > laying about, so nothing I can gdb ... > ... > > send in the output of `grep SHM srclib/apr/include/apr.h` so we can find > out what shared memory subsystems are available on your platform. jail# grep SHM srclib/apr/include/apr.h #define APR_HAVE_SHMEM_MMAP_TMP 1 #define APR_HAVE_SHMEM_MMAP_SHM 1 #define APR_HAVE_SHMEM_MMAP_ZERO1 #define APR_HAVE_SHMEM_SHMGET_ANON 1 #define APR_HAVE_SHMEM_SHMGET 1 #define APR_HAVE_SHMEM_MMAP_ANON1 #define APR_HAVE_SHMEM_BEOS 0 #define APR_USE_SHMEM_MMAP_TMP 0 #define APR_USE_SHMEM_MMAP_SHM 0 #define APR_USE_SHMEM_MMAP_ZERO0 #define APR_USE_SHMEM_SHMGET_ANON 0 #define APR_USE_SHMEM_SHMGET 1 #define APR_USE_SHMEM_MMAP_ANON1 #define APR_USE_SHMEM_BEOS 0 > I tested this code on 5.0-current and I know Justin as well as others > have tested this on earlier versions (4.4 IIRC). jail# uname -a FreeBSD jail 4.5-PRERELEASE FreeBSD 4.5-PRERELEASE #2: Mon Jan 7 11:19:58 AST 2002 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/atelier i386
Re: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?
On Fri, 11 Jan 2002, Justin Erenkrantz wrote: > On Fri, Jan 11, 2002 at 10:27:18AM -0400, Marc G. Fournier wrote: > > > > Morning all ... > > > > Just built a 'jail' environment under a 4.5 system so that I can > > putz around with Apache2-HEAD, where I could "reformat" at will to clean > > up any cruft ... build went great, using: > > > > #!/bin/sh > > ./configure \ > > --prefix=/usr/local/apache \ > > --with-perl=/usr/bin/perl \ > > --enable-so \ > > --with-suexec-caller=nobody \ > > --with-mpm=worker \ > > --enable-threads \ > > --with-port=80 \ > > --enable-mods-shared='all cgid ssl proxy proxy-connect proxy-ftp >proxy-http cache mem-cache file-cache' > > > > Install went as smooth ... but I can't connect ... > > > > logs/error_log shows: > > > > [Fri Jan 11 14:20:17 2002] [info] mod_unique_id: using ip addr 131.162.139.86 > > [Fri Jan 11 14:20:18 2002] [info] mod_unique_id: using ip addr 131.162.139.86 > > [Fri Jan 11 14:20:19 2002] [notice] Digest: generating secret for digest >authentication ... > > [Fri Jan 11 14:20:19 2002] [notice] Digest: done > > [Fri Jan 11 14:20:19 2002] [warn] pid file /usr/local/apache/logs/httpd.pid >overwritten -- Unclean shutdown of previous Apache run? > > [Fri Jan 11 14:20:19 2002] [crit] (78)Function not implemented: Fatal error: could >not open(create) scoreboard > > > > And a ps listing shows no root process, only the one nobody > > process: > > > > jail# ps aux | grep http > > nobody 83618 0.0 1.1 4224 2924 ?? IJ2:20PM 0:00.00 bin/httpd > > > > Even if I comment out the 'ScoreBoard' line in httpd.conf, it > > gives the 'Function not implemented' error ... > > I'm not familiar enough with the "jail," but it sounds like > anonymous mmap may not be supported in this environment. Can you > grep for APR_USE_SHMEM in srclib/apr/include/apr.h and send that? > I'll bet mmap is returning that error. jail# grep APR_USE_SHMEM srclib/apr/include/apr.h #define APR_USE_SHMEM_MMAP_TMP 0 #define APR_USE_SHMEM_MMAP_SHM 0 #define APR_USE_SHMEM_MMAP_ZERO0 #define APR_USE_SHMEM_SHMGET_ANON 0 #define APR_USE_SHMEM_SHMGET 1 #define APR_USE_SHMEM_MMAP_ANON1 #define APR_USE_SHMEM_BEOS 0 > Please understand that threads aren't working on FreeBSD. It's why > you had to manually enable them. =) Thought of that too, so I tried prefork ... got same results ... > However, worker MPM on FreeBSD isn't usable by anyone at this > point other than as a conversation piece. -- justin Actually, I am using on an older 4.4 jail environment: Server Version: Apache/2.0.28 (Unix) mod_ssl/3.0a0 OpenSSL/0.9.6 DAV/2 Current Time: Friday, 11-Jan-2002 17:30:18 GMT Restart Time: Tuesday, 08-Jan-2002 18:28:03 GMT Parent Server Generation: 0 Server uptime: 2 days 23 hours 2 minutes 15 seconds Total accesses: 173 - Total Traffic: 163 kB CPU Usage: u24.5469 s63.2891 cu0 cs0 - .0343% CPU load .000676 requests/sec - 0 B/second - 964 B/request 1 requests currently being processed, 24 idle workers Not a heavily loaded server, mind you ... but it is running ...
Re: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?
Justin, On FreeBSD 4.5 worker is now reasonably usable (albeit in short bursts I will grant), at least it was for me earlier today. We'll have to play with it and see what's going on on other systems and under load though. david - Original Message - From: "Justin Erenkrantz" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 11, 2002 4:08 PM Subject: Re: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing? > On Fri, Jan 11, 2002 at 10:27:18AM -0400, Marc G. Fournier wrote: > > > > Morning all ... > > > > Just built a 'jail' environment under a 4.5 system so that I can > > putz around with Apache2-HEAD, where I could "reformat" at will to clean > > up any cruft ... build went great, using: > > > > #!/bin/sh > > ./configure \ > > --prefix=/usr/local/apache \ > > --with-perl=/usr/bin/perl \ > > --enable-so \ > > --with-suexec-caller=nobody \ > > --with-mpm=worker \ > > --enable-threads \ > > --with-port=80 \ > > --enable-mods-shared='all cgid ssl proxy proxy-connect proxy-ftp proxy-http cache mem-cache file-cache' > > > > Install went as smooth ... but I can't connect ... > > > > logs/error_log shows: > > > > [Fri Jan 11 14:20:17 2002] [info] mod_unique_id: using ip addr 131.162.139.86 > > [Fri Jan 11 14:20:18 2002] [info] mod_unique_id: using ip addr 131.162.139.86 > > [Fri Jan 11 14:20:19 2002] [notice] Digest: generating secret for digest authentication ... > > [Fri Jan 11 14:20:19 2002] [notice] Digest: done > > [Fri Jan 11 14:20:19 2002] [warn] pid file /usr/local/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run? > > [Fri Jan 11 14:20:19 2002] [crit] (78)Function not implemented: Fatal error: could not open(create) scoreboard > > > > And a ps listing shows no root process, only the one nobody > > process: > > > > jail# ps aux | grep http > > nobody 83618 0.0 1.1 4224 2924 ?? IJ2:20PM 0:00.00 bin/httpd > > > > Even if I comment out the 'ScoreBoard' line in httpd.conf, it > > gives the 'Function not implemented' error ... > > I'm not familiar enough with the "jail," but it sounds like > anonymous mmap may not be supported in this environment. Can you > grep for APR_USE_SHMEM in srclib/apr/include/apr.h and send that? > I'll bet mmap is returning that error. > > > Known problem? If not, suggestions on how to debug and provide > > something just a wee bit more useful? I'm not finding any core files > > laying about, so nothing I can gdb ... > > You can do a break in apr_shm_create() and watch the return > value from mmap. I bet it'll be an error condition. > > Please understand that threads aren't working on FreeBSD. It's why > you had to manually enable them. =) > > At the very least, something looks busted with their > pthread_cond_* implementation (or our usage of it) that blows up > our worker scheme. > > If you want to help out with the development of worker and threads > on FreeBSD, you are more than welcome to assist. I would recommend > reading through the various STATUS files for FreeBSD notes. We have > tidbits all over the place. > > However, worker MPM on FreeBSD isn't usable by anyone at this > point other than as a conversation piece. -- justin > >
Re: Apache 2.0.28 memory growth issue.
On Fri, Jan 11, 2002 at 10:22:45AM -0500, Chris Williams wrote: > I am running apache 2.0.28 on Solaris 8 with the worker MPM. I am seeing > memory growth when I use flood to hammer apache when just serving up static > pages. I have had flood running with 250 farmers constantly running the > past hour and the growth of the httpd processes using the ps command has > gone from 4200 to 15000 when using the RSS value from ps. Was this growth > normal in 2.0.28? I know the latest tagged version is 2.0.30 and wonder if > I should try and get it setup on my Solaris box(doesn't have libtool and > other stuff needed to build from CVS). Is/was this a known issue and will > upgrading to a version later than 2.0.28 help? ... Memory leaks and unbounded memory growth in the worker MPM is not something that I've seen nor something that I've heard reported to the list recently. Keep in mind that the multithreaded MPMs will by their very nature have higher memory requirements than their single-threaded counterparts. I regularly see memory usage in the range you've reported. Yesterday while running with 500 threads, I can report that httpd was using 23M (20M RSS). If you consider that 500 prefork processes would take on the order 500*N, where N is usually 3-5MB, then this is a *significant* decrease in memory requirements -- which was one of the major goals of the hybrid multiprocess/ multithreaded httpd model. -aaron
Re: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?
On Fri, Jan 11, 2002 at 10:27:18AM -0400, Marc G. Fournier wrote: ... > [Fri Jan 11 14:20:19 2002] [crit] (78)Function not implemented: Fatal error: could >not open(create) scoreboard ... > Even if I comment out the 'ScoreBoard' line in httpd.conf, it > gives the 'Function not implemented' error ... > > Known problem? If not, suggestions on how to debug and provide > something just a wee bit more useful? I'm not finding any core files > laying about, so nothing I can gdb ... ... send in the output of `grep SHM srclib/apr/include/apr.h` so we can find out what shared memory subsystems are available on your platform. I tested this code on 5.0-current and I know Justin as well as others have tested this on earlier versions (4.4 IIRC). -aaron
Re: Apache 2.0.28 memory growth issue.
Chris Williams wrote: > I am running apache 2.0.28 on Solaris 8 with the worker MPM. I am seeing > memory growth when I use flood to hammer apache when just serving up static > pages. I have had flood running with 250 farmers constantly running the > past hour and the growth of the httpd processes using the ps command has > gone from 4200 to 15000 when using the RSS value from ps. Was this growth > normal in 2.0.28? I know the latest tagged version is 2.0.30 and wonder if > I should try and get it setup on my Solaris box(doesn't have libtool and > other stuff needed to build from CVS). Is/was this a known issue and will > upgrading to a version later than 2.0.28 help? One other question...what is > the correct syntax for the CVS command to get a specific label out of CVS > sat I want to get 2.0.30 instead of the HEAD? > Thanks! > Chris > > Hi Chris. I'm not sure about .28, but .29 (+aprtableget patch from 30) seems pretty stable for us, and we haven't seen that memory growth you describe. does the RSS size constantly grow from the start, or does it sit still for a period of time and then start increasing. also it could be a module you are running.
Re: cvs commit: apache-1.3/src/main http_core.c
"William A. Rowe, Jr." wrote: > > From: "Roy T. Fielding" <[EMAIL PROTECTED]> > Sent: Thursday, January 10, 2002 5:47 PM > > And no features should be added to 1.3 without the parallel patch to 2.0, > if it is at all relevant. Ken, is there a patch forthcoming? You betcha. Docco too. :-) Okey, MMN bump being changed to minor.. -- #kenP-)} Ken Coar, Sanagendamgagwedweinini http://Golux.Com/coar/ Author, developer, opinionist http://Apache-Server.Com/ "All right everyone! Step away from the glowing hamburger!"
Re: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?
On Fri, Jan 11, 2002 at 10:27:18AM -0400, Marc G. Fournier wrote: > > Morning all ... > > Just built a 'jail' environment under a 4.5 system so that I can > putz around with Apache2-HEAD, where I could "reformat" at will to clean > up any cruft ... build went great, using: > > #!/bin/sh > ./configure \ > --prefix=/usr/local/apache \ > --with-perl=/usr/bin/perl \ > --enable-so \ > --with-suexec-caller=nobody \ > --with-mpm=worker \ > --enable-threads \ > --with-port=80 \ > --enable-mods-shared='all cgid ssl proxy proxy-connect proxy-ftp proxy-http >cache mem-cache file-cache' > > Install went as smooth ... but I can't connect ... > > logs/error_log shows: > > [Fri Jan 11 14:20:17 2002] [info] mod_unique_id: using ip addr 131.162.139.86 > [Fri Jan 11 14:20:18 2002] [info] mod_unique_id: using ip addr 131.162.139.86 > [Fri Jan 11 14:20:19 2002] [notice] Digest: generating secret for digest >authentication ... > [Fri Jan 11 14:20:19 2002] [notice] Digest: done > [Fri Jan 11 14:20:19 2002] [warn] pid file /usr/local/apache/logs/httpd.pid >overwritten -- Unclean shutdown of previous Apache run? > [Fri Jan 11 14:20:19 2002] [crit] (78)Function not implemented: Fatal error: could >not open(create) scoreboard > > And a ps listing shows no root process, only the one nobody > process: > > jail# ps aux | grep http > nobody 83618 0.0 1.1 4224 2924 ?? IJ2:20PM 0:00.00 bin/httpd > > Even if I comment out the 'ScoreBoard' line in httpd.conf, it > gives the 'Function not implemented' error ... I'm not familiar enough with the "jail," but it sounds like anonymous mmap may not be supported in this environment. Can you grep for APR_USE_SHMEM in srclib/apr/include/apr.h and send that? I'll bet mmap is returning that error. > Known problem? If not, suggestions on how to debug and provide > something just a wee bit more useful? I'm not finding any core files > laying about, so nothing I can gdb ... You can do a break in apr_shm_create() and watch the return value from mmap. I bet it'll be an error condition. Please understand that threads aren't working on FreeBSD. It's why you had to manually enable them. =) At the very least, something looks busted with their pthread_cond_* implementation (or our usage of it) that blows up our worker scheme. If you want to help out with the development of worker and threads on FreeBSD, you are more than welcome to assist. I would recommend reading through the various STATUS files for FreeBSD notes. We have tidbits all over the place. However, worker MPM on FreeBSD isn't usable by anyone at this point other than as a conversation piece. -- justin
Re: [PATCH] fix segfault in core input filter
On Fri, Jan 11, 2002 at 09:03:36AM -0500, Jeff Trawick wrote: > With current CVS code, prefork, APR_POOL_DEBUG, Linux, and > ElectricFence I readily hit a segfault on the APR_BRIGADE_NORMALIZE() > call in core input filter. It segfaulted because the brigade was > empty. With the following patch I can't seem to hit a segfault > anymore. > > Perhaps the brigade shouldn't be empty for some reason... If the brigade in core_input_filter is empty, it means that the socket (and its bucket) has disappeared on us. Most likely, someone isn't paying attention to APR_EOF return values. We should return an error or at the very least APR_EOF if we see an empty brigade there. But, it would be more useful to find out which input filter isn't paying attention. -- justin
Re: cvs commit: httpd-2.0 STATUS
I was seeing the segraults in a different location on Solaris, probably because my testing method was different. I created idle connections to the httpd to use up all the descriptors, rather than a flood of real requests. So we probably had (at least) two different bugs related to running out of file descriptors. --Brian Jeff Trawick wrote: >[EMAIL PROTECTED] writes: > >>brianp 02/01/11 00:07:07 >> >> Modified:.STATUS >> Log: >> Updated STATUS to cover the worker segfault fixes >> >> > >> -* The worker MPM on Solaris segfaults when it runs out of file >> - descriptors. (This may affect other MPMs and/or platforms.) >> > >I can still readily hit this on current code (the same code that no >longer segfaults with graceful restart). > >[Fri Jan 11 07:26:37 2002] [error] (24)Too many open files: >apr_accept: (client socket) >[Fri Jan 11 07:26:37 2002] [error] [client 127.0.0.1] (24)Too many >open files: file permissions deny server access: /exp >ort/home/trawick/apacheinst/htdocs/index.html.en >[Fri Jan 11 07:26:37 2002] [error] [client 127.0.0.1] (24)Too many >open files: cannot access type map file: /export/home >/trawick/apacheinst/error/HTTP_FORBIDDEN.html.var >[Fri Jan 11 07:26:38 2002] [notice] child pid 25493 exit signal >Segmentation fault (11), possible coredump in /export/ho >me/trawick/apacheinst > >This is the same coredump I saw before: > >#0 0xff33a3cc in apr_wait_for_io_or_timeout (sock=0x738360, > for_read=1) at sendrecv.c:70 >70 FD_SET(sock->socketdes, &fdset); > >The socket has already been closed so trying to set bit -1 segfaults. >
Apache 2.0.28 memory growth issue.
I am running apache 2.0.28 on Solaris 8 with the worker MPM. I am seeing memory growth when I use flood to hammer apache when just serving up static pages. I have had flood running with 250 farmers constantly running the past hour and the growth of the httpd processes using the ps command has gone from 4200 to 15000 when using the RSS value from ps. Was this growth normal in 2.0.28? I know the latest tagged version is 2.0.30 and wonder if I should try and get it setup on my Solaris box(doesn't have libtool and other stuff needed to build from CVS). Is/was this a known issue and will upgrading to a version later than 2.0.28 help? One other question...what is the correct syntax for the CVS command to get a specific label out of CVS sat I want to get 2.0.30 instead of the HEAD? Thanks! Chris
Re: cvs commit: httpd-2.0/server/mpm/worker worker.c
Is this the same issue I just reported with: "Subject: [HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?" If so, then there is another bug in there somewhere, since I just checked, and I'm running v1.60 of worker.c: revision 1.60 date: 2002/01/11 08:01:11; author: brianp; state: Exp; lines: +1 -1 Fix for a segfault in the worker MPM during graceful shutdown: The per-transaction pools in the worker MPM can't be children of the listener thread's pool, because that pool may go out of scope while some workers are still procesing requests using the transaction pools. jail# cvs status worker.c === File: worker.c Status: Up-to-date Working revision:1.60 Repository revision: 1.60/home/cvspublic/httpd-2.0/server/mpm/worker/worker.c,v Sticky Tag: (none) Sticky Date: (none) Sticky Options: (none) On Fri, 11 Jan 2002, David Reid wrote: > This also fixes the shutdown issue with worker MPM on FreeBSD which is good > :) > > david > > > > brianp 02/01/11 00:01:11 > > > > Modified:.CHANGES > >server/mpm/worker worker.c > > Log: > > Fix for a segfault in the worker MPM during graceful shutdown: > > The per-transaction pools in the worker MPM can't be children of > > the listener thread's pool, because that pool may go out of scope > > while some workers are still procesing requests using the transaction > > pools. > > > > >
[HEAD] --with-mpm=worker under FreeBSD 4.5 does nothing?
Morning all ... Just built a 'jail' environment under a 4.5 system so that I can putz around with Apache2-HEAD, where I could "reformat" at will to clean up any cruft ... build went great, using: #!/bin/sh ./configure \ --prefix=/usr/local/apache \ --with-perl=/usr/bin/perl \ --enable-so \ --with-suexec-caller=nobody \ --with-mpm=worker \ --enable-threads \ --with-port=80 \ --enable-mods-shared='all cgid ssl proxy proxy-connect proxy-ftp proxy-http cache mem-cache file-cache' Install went as smooth ... but I can't connect ... logs/error_log shows: [Fri Jan 11 14:20:17 2002] [info] mod_unique_id: using ip addr 131.162.139.86 [Fri Jan 11 14:20:18 2002] [info] mod_unique_id: using ip addr 131.162.139.86 [Fri Jan 11 14:20:19 2002] [notice] Digest: generating secret for digest authentication ... [Fri Jan 11 14:20:19 2002] [notice] Digest: done [Fri Jan 11 14:20:19 2002] [warn] pid file /usr/local/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run? [Fri Jan 11 14:20:19 2002] [crit] (78)Function not implemented: Fatal error: could not open(create) scoreboard And a ps listing shows no root process, only the one nobody process: jail# ps aux | grep http nobody 83618 0.0 1.1 4224 2924 ?? IJ2:20PM 0:00.00 bin/httpd Even if I comment out the 'ScoreBoard' line in httpd.conf, it gives the 'Function not implemented' error ... Known problem? If not, suggestions on how to debug and provide something just a wee bit more useful? I'm not finding any core files laying about, so nothing I can gdb ... Oh, I haven't compiled anything into it other then the base source code (ie. no php or mod_perl or anything ... just straight apache right now) ... am going to try out a prefork version next ...
[PATCH] fix segfault in core input filter
With current CVS code, prefork, APR_POOL_DEBUG, Linux, and
ElectricFence I readily hit a segfault on the APR_BRIGADE_NORMALIZE()
call in core input filter. It segfaulted because the brigade was
empty. With the following patch I can't seem to hit a segfault
anymore.
Perhaps the brigade shouldn't be empty for some reason...
If this is really the right issue, APR_BRIGADE_NORMALIZE() should
probably have the logic to check for an empty brigade.
Index: server/core.c
===
RCS file: /home/cvs/httpd-2.0/server/core.c,v
retrieving revision 1.128
diff -u -r1.128 core.c
--- server/core.c 8 Jan 2002 17:07:19 - 1.128
+++ server/core.c 11 Jan 2002 13:57:30 -
@@ -2879,7 +2879,9 @@
}
/* ### This is bad. */
-APR_BRIGADE_NORMALIZE(ctx->b);
+if (!APR_BRIGADE_EMPTY(ctx->b)) {
+APR_BRIGADE_NORMALIZE(ctx->b);
+}
/* ### AP_MODE_PEEK is a horrific name for this mode because we also
* eat any CRLFs that we see. That's not the obvious intention of
--
Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: cvs commit: httpd-2.0/server/mpm/worker worker.c
This also fixes the shutdown issue with worker MPM on FreeBSD which is good :) david > brianp 02/01/11 00:01:11 > > Modified:.CHANGES >server/mpm/worker worker.c > Log: > Fix for a segfault in the worker MPM during graceful shutdown: > The per-transaction pools in the worker MPM can't be children of > the listener thread's pool, because that pool may go out of scope > while some workers are still procesing requests using the transaction > pools. >
Re: mod_include???
OK, all seems to have returned to normality, though I have absolutely no idea how/why... david - Original Message - From: "David Reid" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 11, 2002 12:40 PM Subject: Re: mod_include??? > FWIW, I've just done an update and build on beos and have the same problem, > this time with mod_access. Either I've got something screwy in my build > tree's or Houston, we have a problem. > > david > > - Original Message - > From: "David Reid" <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]> > Sent: Friday, January 11, 2002 10:53 AM > Subject: Re: mod_include??? > > > > Following Will's advise to do a make clean;make failed to correct the > > problem. > > > > Here is what i did... > > > > cvs update -dP > > ./buildconf > > ./configure ... > > make clean;make > > rm -rf /usr/local/apache2/* > > make install > > cd /usr/local/apache2/bin > > ./apachectl start > > > > And the result? > > > > httpd: module "mod_include.c" is not compatible with this version of > Apache. > > Please contact the vendor for the correct version. > > > > This has only been happenning since I last updated (3 days ago). > > > > david > > > > - Original Message - > > From: "David Reid" <[EMAIL PROTECTED]> > > To: <[EMAIL PROTECTED]> > > Sent: Friday, January 11, 2002 9:27 AM > > Subject: mod_include??? > > > > > > > Just done a cvs update/buildconf/make/make install and I'm seeing an > error > > > message that mod_include.c is not compatible with this version of > apache! > > > > > > Anyone any ideas? > > > > > > david > > > > > > > > > > > >
Re: pool problems with HEAD??
Jeff Trawick <[EMAIL PROTECTED]> writes:
> . Current code (worker MPM on Linux, at least) will segfault like
> crazy if you turn on APR_POOL_DEBUG and add "-lefence " to the
> beginning of EXTRA_LIBS in config_vars.mk (I guess this is the
> right way to use ElectricFence).
After sufficient sleep and/or caffiene, it occurred to me that I
should try a non-threaded build with efence and APR_POOL_DEBUG on this
Linux box. Sure enough, plentiful segfaults, but more importantly
usable coredumps...
I just fixed one of the causes of segfaults (bad parm in mod_rewrite)
(another misstep on my part though... in the hurry to try to find
the problem in different environments I neglected to modernize this
config file... but at least we got a fix out of it :) ).
I'm still getting this segfault:
#0 0x80ceb33 in core_input_filter (f=0x40af7fec, b=0x40c13ff4,
mode=AP_MODE_BLOCKING,
readbytes=0xb9c8) at core.c:2882
2882APR_BRIGADE_NORMALIZE(ctx->b);
(gdb) where
#0 0x80ceb33 in core_input_filter (f=0x40af7fec, b=0x40c13ff4,
mode=AP_MODE_BLOCKING,
readbytes=0xb9c8) at core.c:2882
#1 0x80c56d9 in ap_get_brigade (next=0x40af7fec, bb=0x40c13ff4,
mode=AP_MODE_BLOCKING,
readbytes=0xb9c8) at util_filter.c:362
#2 0x80cea2b in net_time_filter (f=0x40c2dfec, b=0x40c13ff4,
mode=AP_MODE_BLOCKING,
readbytes=0xb9c8) at core.c:2844
#3 0x80c56d9 in ap_get_brigade (next=0x40c2dfec, bb=0x40c13ff4,
mode=AP_MODE_BLOCKING,
readbytes=0xb9c8) at util_filter.c:362
#4 0x80c6c38 in ap_rgetline (s=0x40b1fe9c, n=8192, r=0x40b1fe84,
fold=0) at protocol.c:225
#5 0x80c7227 in read_request_line (r=0x40b1fe84) at protocol.c:429
#6 0x80c789b in ap_read_request (conn=0x40aabfb8) at protocol.c:608
#7 0x807a2f9 in ap_process_http_connection (c=0x40aabfb8) at
http_core.c:273
#8 0x80c2f1d in ap_run_process_connection (c=0x40aabfb8) at
connection.c:84
#9 0x80c335d in ap_process_connection (c=0x40aabfb8) at
connection.c:230
#10 0x80b49e0 in child_main (child_num_arg=28) at prefork.c:716
#11 0x80b4b79 in make_child (s=0x41af9f9c, slot=28) at prefork.c:806
#12 0x80b4e74 in perform_idle_server_maintenance (p=0x4021afd8) at
prefork.c:947
#13 0x80b52be in ap_mpm_run (_pconf=0x4021afd8, plog=0x40797fd8,
s=0x41af9f9c)
at prefork.c:1118
#14 0x80bbd8e in main (argc=1, argv=0xbca4) at main.c:461
(gdb) p ctx
$1 = (core_ctx_t *) 0x40c43ffc
(gdb) p *ctx
$2 = {b = 0x40c49ff4}
(gdb)
This is fairly close to one of the segfaults I was trying to find
yesterday when I ended up going down other paths due to other bugs in
the server and due to missteps of my own.
That segfault has a similar traceback to the one shown above except
that it dies on this apr_bucket_read():
/* we are reading a single LF line, e.g. the HTTP headers */
while (!APR_BRIGADE_EMPTY(ctx->b)) {
const char *pos;
e = APR_BRIGADE_FIRST(ctx->b);
rv = apr_bucket_read(e, &str, &len, mode);
/* We should treat EAGAIN here the same as we do for EOF
(brigade is
have coredump, will debug...
--
Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: mod_include???
FWIW, I've just done an update and build on beos and have the same problem, this time with mod_access. Either I've got something screwy in my build tree's or Houston, we have a problem. david - Original Message - From: "David Reid" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 11, 2002 10:53 AM Subject: Re: mod_include??? > Following Will's advise to do a make clean;make failed to correct the > problem. > > Here is what i did... > > cvs update -dP > ./buildconf > ./configure ... > make clean;make > rm -rf /usr/local/apache2/* > make install > cd /usr/local/apache2/bin > ./apachectl start > > And the result? > > httpd: module "mod_include.c" is not compatible with this version of Apache. > Please contact the vendor for the correct version. > > This has only been happenning since I last updated (3 days ago). > > david > > - Original Message - > From: "David Reid" <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]> > Sent: Friday, January 11, 2002 9:27 AM > Subject: mod_include??? > > > > Just done a cvs update/buildconf/make/make install and I'm seeing an error > > message that mod_include.c is not compatible with this version of apache! > > > > Anyone any ideas? > > > > david > > > > > >
Re: cvs commit: httpd-2.0 STATUS
[EMAIL PROTECTED] writes: > brianp 02/01/11 00:07:07 > > Modified:.STATUS > Log: > Updated STATUS to cover the worker segfault fixes > > -* The worker MPM on Solaris segfaults when it runs out of file > - descriptors. (This may affect other MPMs and/or platforms.) I can still readily hit this on current code (the same code that no longer segfaults with graceful restart). [Fri Jan 11 07:26:37 2002] [error] (24)Too many open files: apr_accept: (client socket) [Fri Jan 11 07:26:37 2002] [error] [client 127.0.0.1] (24)Too many open files: file permissions deny server access: /exp ort/home/trawick/apacheinst/htdocs/index.html.en [Fri Jan 11 07:26:37 2002] [error] [client 127.0.0.1] (24)Too many open files: cannot access type map file: /export/home /trawick/apacheinst/error/HTTP_FORBIDDEN.html.var [Fri Jan 11 07:26:38 2002] [notice] child pid 25493 exit signal Segmentation fault (11), possible coredump in /export/ho me/trawick/apacheinst This is the same coredump I saw before: #0 0xff33a3cc in apr_wait_for_io_or_timeout (sock=0x738360, for_read=1) at sendrecv.c:70 70 FD_SET(sock->socketdes, &fdset); The socket has already been closed so trying to set bit -1 segfaults. -- Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site: http://www.geocities.com/SiliconValley/Park/9289/ Born in Roswell... married an alien...
Re: FreeBSD load statistics.
On Fri, Jan 11, 2002 at 12:31:28AM -0800, Brian Pane wrote: > The other way to isolate the problem would be to get a snapshot of > what the kernel is doing on daedalus during these spikes. Is there > a way to do that in FreeBSD? The only debugging tool I know of is ktrace. But, that only shows the syscalls made by the program, not what the kernel is doing internally. This is why Solaris is so nice - pstack and /usr/proc/bin/* progs can tell you what the kernel itself is doing. Furthermore, the kernel debuggers are only available locally and stop the machine, so I don't think that's an option for daedalus since its in a NOC that none of us have access to. Anyone else have ideas or suggestions? Brian (FirstBrian, that is) or Manoj might have some ideas here. -- justin
Re: cvs commit: httpd-2.0/server/mpm/worker worker.c
[EMAIL PROTECTED] writes: > brianp 02/01/11 00:01:11 > > Modified:.CHANGES >server/mpm/worker worker.c > Log: > Fix for a segfault in the worker MPM during graceful shutdown: > The per-transaction pools in the worker MPM can't be children of > the listener thread's pool, because that pool may go out of scope > while some workers are still procesing requests using the transaction > pools. Thanks! I can no longer hit the graceful restart segfault on either AIX or Solaris. -- Jeff Trawick | [EMAIL PROTECTED] | PGP public key at web site: http://www.geocities.com/SiliconValley/Park/9289/ Born in Roswell... married an alien...
[ANNOUNCE] OpenPKG 1.0 (cross-platform RPM-based Unix software packaging)
Please excuse this slightly off-topic posting. People have wondered what I've done over the last 14 months and why further development on mod_ssl and my engagement in the OpenSSL and Apache projects had to be slowed down in this time. Most of my contributions were moved to the silent background. Some people even feared that Ralf is doing closed source software development now. No, the opposite is true. I'm still the same Open Source fanatic, believe me. I just avoided to make something public until is is really ready for the public. But since today the result of over one year of strong development is now publically available and I'm proudly announcing it: OpenPKG, the world of cross-platform RPM-based Unix software packaging. OpenPKG is one of the largest projects I've ever done in my life. My development team at Cable & Wireless Germany plus dozens of contributors have helped together since November 2000 to solve a problem I already tried to solve two times in the last 10 years. And the third solution OpenPKG is the one which finally successfully solved it (at least for me ;) We are using it in production since April 2001 to establish all customer servers in the Internet Solution Center (ISC) of Cable & Wireless Germany and certainly there will be some other groups in this world which benefit from our work, too. That's why it is both based on Open Source software and itself released again as Open Source. If you want to know more, glance over the attached PR text and visit our project site at http://www.openpkg.org/. Thanks for listening and... happy packaging! Oh, and we have also packaged OpenSSL and Apache+modssl in OpenPKG, of course ;) Yours, Ralf S. Engelschall [EMAIL PROTECTED] www.engelschall.com = Now available: OpenPKG 1.0 www.openpkg.org A flexible and powerful software packaging facility, OpenPKG eases installation and administration of Unix software across several platforms. It primarily targets the Unix platforms FreeBSD, Linux and Solaris, but is portable across mostly all modern Unix flavors. Consolidating different vendor approaches into a unified architecture, it assists in administration of large networks previously complicated by nonconformant systems. OpenPKG leverages proven technologies like Red Hat Package Manager (RPM) and neatly provides an additional system layer on top of the operating system. It is a fully self-contained with minimal external dependencies (no RPM pre-installation required), and installs itself by means of a tricky bootstrapping procedure with minimal operating system intrusion. OpenPKG especially supports multiple installation instances on the same system. OpenPKG was created in November 2000 and after over one year of development it is already a mature technology in production use. It is available as Open Source and is further maintained by both Ralf S. Engelschall's development team at Cable & Wireless Germany and their contributors. For more details visit: http://www.openpkg.org/ ftp://ftp.openpkg.org/ =
Re: mod_include???
Following Will's advise to do a make clean;make failed to correct the problem. Here is what i did... cvs update -dP ./buildconf ./configure ... make clean;make rm -rf /usr/local/apache2/* make install cd /usr/local/apache2/bin ./apachectl start And the result? httpd: module "mod_include.c" is not compatible with this version of Apache. Please contact the vendor for the correct version. This has only been happenning since I last updated (3 days ago). david - Original Message - From: "David Reid" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 11, 2002 9:27 AM Subject: mod_include??? > Just done a cvs update/buildconf/make/make install and I'm seeing an error > message that mod_include.c is not compatible with this version of apache! > > Anyone any ideas? > > david > >
Re: mod_include???
Bumped the MMN for big scoreboard changes. Score no longer contains that global, server, worker pointers bit, that's all private now. The scoreboard global member became a pointer into the beginning of the scoreboard, and scores are now relocation-safe. Bill - Original Message - From: "David Reid" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 11, 2002 3:27 AM Subject: mod_include??? > Just done a cvs update/buildconf/make/make install and I'm seeing an error > message that mod_include.c is not compatible with this version of apache! > > Anyone any ideas? > > david > >
Re: [PATCH] 1.3: further Cygwin platform support
On Fri, Jan 11, 2002 at 10:37:12AM +0100, Stipe Tolj wrote: > Hi Martin, > > > > > Testing for [ ! -f "$src." ] is not the same thing > > as testing for [ ! -f "$src" ]. > > I know! -- somehow I have to trick the beast on Cygwin to get a clean > make install to copy the whole executables. > > Should I add a uname case switch here to be on the same side?! > Or maybe a checking of env var $OSTYPE?! Yes, either would be ok with me. OSTYPE is probably cheaper ;-) > Wie wär's mit einem Treffen auf der CeBIT? Schätze werden einige von > uns da sein?! Sorry, I think no :-( Martin -- <[EMAIL PROTECTED]> | Fujitsu Siemens Fon: +49-89-636-46021, FAX: +49-89-636-47655 | 81730 Munich, Germany
Re: [PATCH] 1.3: further Cygwin platform support
Hi Martin, > Sorry to be picky here (Slowly, you're beginning to convince > me that the extra test for "! -f $src" is not worth the trouble > of fighting against cygwin's stupidity)... > > Testing for [ ! -f "$src." ] is not the same thing > as testing for [ ! -f "$src" ]. I know! -- somehow I have to trick the beast on Cygwin to get a clean make install to copy the whole executables. Should I add a uname case switch here to be on the same side?! Or maybe a checking of env var $OSTYPE?! Have you any other proposal for getting this *done*? > And it is the latter which I was worried about: > what happens if -on a unix platform- I happen > to have a $src.exe in my directory? Yes, Apache will assume > that my platform uses the .exe extension. And it will break. That's true -- but, there is no Unix platform (at least currently) that *build* httpd.exe (or any other .exe extension executable), is there?! > But if it breaks on Cygwin, I am much less worried that when > it breaks on unix, to be honest. So IMHO, it is cygwin that > should be fixed so that it will allow us to make such a test. I agree and I'll try to propose a change to that. But the stupidity was introduced to demands the Cygwin folks had, so I'm not that eager that the change will be done on that side very easily. Wie wär's mit einem Treffen auf der CeBIT? Schätze werden einige von uns da sein?! Stipe [EMAIL PROTECTED] --- Wapme Systems AG Münsterstr. 248 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
mod_include???
Just done a cvs update/buildconf/make/make install and I'm seeing an error message that mod_include.c is not compatible with this version of apache! Anyone any ideas? david
Re: FreeBSD load statistics.
Justin Erenkrantz wrote: >FWIW, Aaron was telling me that he remembers seeing high run-queue >lengths on machines with failing hard drives. Can anyone reproduce >this behavior (high run-queue with idle CPU) on any other machines? >Perhaps we could perform some testing on icarus (which should be a >daedalus clone). > The other way to isolate the problem would be to get a snapshot of what the kernel is doing on daedalus during these spikes. Is there a way to do that in FreeBSD? --Brian
Fw: Current CVS on Win32
From: "Sebastian Bergmann" <[EMAIL PROTECTED]>
Sent: Friday, January 11, 2002 1:33 AM
> "William A. Rowe, Jr." wrote:
> > ScoreboardFile logs/apache.score should work fine.
>
> Yep, doesn't crash now.
Ok... consider this a bug, no default ScoreboardFile name. One that
I don't have time to trace right now. The errors lie in;
apr_shm_{create|attach} cleanup - may not have a filename [anon].
win32 scoreboard should _not_ use anon with no scoreboard name, it
should revert to the old method if scoreboard name is null (use the
NOT_SHARED bit that we do in -X). At this moment Win32 doesn't
-need- to have a shared score, until we create multiple processes,
or unless a module in the parent wants scoreboard data from the child.
If such a module is installed [for the moment] they aught to configure
the ScoreboardFile directive.
Simple patch for anyone looking for a mini-project. I'll fix soon if
noone steps up, but those of you hacking WinNT might need to set your
ScoreboardFile directive, as a workaround for right now.
Second Win32 bugaboo at this moment;
mod_rewrite's locks were messed up by the recent changes to the
apr_lock_t api (which we were deprecating in the first place, so
I'm far from certain why it changed.) It has to do with creating
the rewrite logs mutex. Either don't load mod_rewrite, or give
the rewrite log mutex a name {dunno what that syntax is, offhand.}
The solution is to find the brokenness in apr_lock_t, but if it goes
away, why waste the time?
Of course, to deprecate apr_lock_t, I see that we still need to add
an apr_procthread_lock (LOCKALL), since Aaron informs me that on some
Unicies, thread locks don't work cross-proc, and proc locks won't work
cross-threads. Geesh.
I refuse to look up the directive, because I refuse to fix the problem
on my machine till the bug is licked. ITMT, I've run out of time chasing
that mutex destruction in ssl to hack any further; won't have alot of
Apache time on my hands till midweek next week, maybe even next weekend.
Bill
Re: cvs commit: httpd-2.0/server/mpm/worker worker.c
[EMAIL PROTECTED] wrote: >brianp 02/01/11 00:01:11 > > Modified:.CHANGES > server/mpm/worker worker.c > Log: > Fix for a segfault in the worker MPM during graceful shutdown: > The per-transaction pools in the worker MPM can't be children of > the listener thread's pool, because that pool may go out of scope > while some workers are still procesing requests using the transaction > pools. > This should fix both segfaults that Jeff reported on Solaris: the graceful shutdown one, and the out-of-file-descriptors one. --Brian
