On Fri, Jun 05, 2015 at 05:15:09PM -0500, Matthew Martin wrote:
> >Synopsis: Serving large files with httpd eats fscktons of memory.
> >Category: system
> >Environment:
> System : OpenBSD 5.7
> Details : OpenBSD 5.7 (GENERIC) #0: Thu Apr 30 22:01:01 CEST 2015
>
> r...@stable-57-amd64.mtier.org:/binpatchng/work-binpatch57-amd64/src/sys/arch/amd64/compile/GENERIC
>
> Architecture: OpenBSD.amd64
> Machine : amd64
> >Description:
> A couple of people concurrently downloading iso sized files can drag
> the server into using all of swap (there's only a gig) in a
> frighteningly short time.
>
> Revision 1.30 of
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/httpd/server_file.c
> had message "Adjust the read/write watermarks according to the TCP
> send buffer. This fixes sending of large files. Previously, httpd was
> reading the input file too quickly and could run out of memory when
> filling the input buffer."
>
> That pretty well describes the situation I'm seeing now with 5.7
> (-RELEASE + m:tier binpatches). It also happens with -CURRENT (as of
> Jun 5) httpd compiled and slapped in its place.
>
> >How-To-Repeat:
> Something along the lines of
> hostA# dd if=/dev/zero of=/var/www/htdocs/test_512.dat bs=1M count=512
> hostB$ ftp http://hostA/test_512.dat
>
> >Fix:
> I've attempted various settings of 'tcp socket buffer' and shotgunning
> around the lines modified in 1.30 to no avail. Trying to limit its
> consumption via login.conf just makes it die when it hits the limit.
the problem is that we have to bufferevents. clt->clt_bev which is
handleing the accept(2)ed socket and clt->clt_srvbev which is
handleing the fd of the served file.
it goes like this:
clt_bev: Here I am, brain the size of a planet, and they ask me to serve
a file.
clt_srvbev: Whee, I can read clt_sndbufsiz bytes from the file. Hello
clt_bev, here, have some data.
clt_bev: Ok, I can't send it right now, but I have a buffer, I'll put
the data at the end.
[last message repeated n times]
clt_bev: Phew, I can send some data to the client, hopfully clt_bev
stops to send me data soon.
and so on... In the end the whole iso ends up in clt_bev.
I'm not sure yet about the 32 and 256 magic numbers, they seem to it
some kind of sweet spot on *my* system. Make them to small and
everything slows down, make them to large and you waste memory.
Please try this:
diff --git server.c server.c
index ca67a47..22b47b8 100644
--- server.c
+++ server.c
@@ -704,7 +704,7 @@ server_input(struct client *clt)
/* Adjust write watermark to the socket buffer output size */
bufferevent_setwatermark(clt->clt_bev, EV_WRITE,
- clt->clt_sndbufsiz, 0);
+ 32 * clt->clt_sndbufsiz, 0);
/* Read at most amount of data that fits in one fcgi record. */
bufferevent_setwatermark(clt->clt_bev, EV_READ, 0, FCGI_CONTENT_SIZE);
@@ -729,6 +729,10 @@ server_write(struct bufferevent *bev, void *arg)
goto done;
bufferevent_enable(bev, EV_READ);
+
+ if (clt->clt_srvbev && !(clt->clt_srvbev->enabled & EV_READ))
+ bufferevent_enable(clt->clt_srvbev, EV_READ);
+
return;
done:
(*bev->errorcb)(bev, EVBUFFER_WRITE|EVBUFFER_EOF, bev->cbarg);
@@ -769,6 +773,11 @@ server_read(struct bufferevent *bev, void *arg)
goto fail;
if (clt->clt_done)
goto done;
+
+ if (EVBUFFER_LENGTH(EVBUFFER_OUTPUT(clt->clt_bev)) > 256 *
+ clt->clt_sndbufsiz)
+ bufferevent_disable(bev, EV_READ);
+
return;
done:
(*bev->errorcb)(bev, EVBUFFER_READ|EVBUFFER_EOF, bev->cbarg);
--
I'm not entirely sure you are real.
