[Resending because apparently it didn't get through yesterday.]
Hi,
my HTTP server thingy came to a screeching halt when serving many
large files, so I added a form of Linux sendfile support that requires
just a few changes and supports some interesting uses (e.g. sending
the HTTP headers normally but the HTTP body via sendfile).
The server now runs very fast, with very little CPU and RAM. This is
preliminary, probably buggy, and I don't understand the evbuffer
functionality very well (e.g. draining).
In addition to the normal data evbuffers, this patch adds two types of
evbuffers: (1) sendfile buffers, and (2) data buffers to which a
sendfile buffer has been added. This distinction is recorded in the
sf_fd field. There are two additonal fields, sf_off and sf_buf.
*** libevent-1.4.3-stable/event.h 2008-02-23 02:36:12.000000000 +0100
--- libevent-1.4.3-stable-sendfile/event.h 2008-05-07
15:40:32.000000000 +0200
***************
*** 724,729 ****
--- 724,746 ----
size_t totallen;
size_t off;
+ /*
+ sf_fd == -1. A standard data buffer. sf_off and sf_buf are
+ unused.
+
+ sf_fd == -2. A buffer to which a sendfile buffer has been
+ added at the end (sf_buf). off holds the combined size of
+ the RAM data of this buffer and the added sendfile buffer.
+ sf_off is unused.
+
+ sf_fd >= 0. A pure sendfile buffer. sf_fd, sf_off, and off
+ are the arguments in_fd, offset, and count to sendfile(),
+ respectively. sf_buf is unused.
+ */
+ int sf_fd;
+ off_t sf_off;
+ struct evbuffer *sf_buf;
+
void (*cb)(struct evbuffer *, size_t, size_t, void *);
void *cbarg;
};
The adding and writing of buffers (evbuffer_add_buffer,
evbuffer_write) is changed:
To keep it simple this system only allows adding a sendfile buffer to
a data buffer. So, the sendfile buffer must be the last step in the
chain, which works fine for e.g. many HTTP uses cases. You cannot add
a sendfile buffer to another sendfile buffer, or add a data buffer to
a sendfile buffer. I think these things would theoretically be
possible, but require a lot more housekeeping than the current
solution, which plugs directly into the existing draining code.
When a sendfile buffer is added after a data buffer, we trick and
increment the data buffer's length by the length of the sendfile
buffer, without actually copying the data to the buffer (which would
completely defeat the purpose of sendfile, of course). Then, later
when it comes to writing this combined buffer, we just need to check
whether all "normal" data has been written, and then switch over to
writing the sendfile data. This *seems* to work nicely but I am not
100% sure that every situation is handled OK.
*** libevent-1.4.3-stable/buffer.c 2007-11-12 03:37:32.000000000 +0100
--- libevent-1.4.3-stable-sendfile/buffer.c 2008-05-07
16:36:44.000000000 +0200
***************
*** 61,66 ****
--- 61,68 ----
#include <unistd.h>
#endif
+ #include <sys/sendfile.h>
+
#include "event.h"
#include "config.h"
***************
*** 71,76 ****
--- 73,81 ----
buffer = calloc(1, sizeof(struct evbuffer));
+ /* Make all buffers non-sendfile buffers initially. */
+ buffer->sf_fd = -1;
+
return (buffer);
}
***************
*** 100,105 ****
--- 105,130 ----
{
int res;
+ if (outbuf->sf_fd == -1) {
+ /* The output buffer is a data buffer. */
+ if (inbuf->sf_fd >= 0) {
+ /* The input buffer is a sendfile buffer.
+ Mark this buffer as containing a sendfile
+ buffer and "virtually" extend the buffer by
+ the size of the sendfile buffer. */
+ outbuf->sf_fd = -2;
+ outbuf->sf_buf = inbuf;
+ outbuf->off += inbuf->off;
+ return 0;
+ }
+ } else {
+ /* No support for nested or concatenated sendfile
+ buffers, i.e. if the output buffer is or contains a
+ sendfile buffer, we cannot add another sendfile
+ buffer to it. */
+ return -1;
+ }
+
/* Short cut for better performance */
if (outbuf->off == 0) {
struct evbuffer tmp;
***************
*** 415,421 ****
int n;
#ifndef WIN32
! n = write(fd, buffer->buffer, buffer->off);
#else
n = send(fd, buffer->buffer, buffer->off, 0);
#endif
--- 440,467 ----
int n;
#ifndef WIN32
! switch(buffer->sf_fd) {
! case -1:
! /* A data buffer. */
! n = write(fd, buffer->buffer, buffer->off);
! break;
! case -2:
! /* A data buffer with a sendfile buffer at the end.
! First write the actual data in the buffer, then
! switch to the sendfile data. */
! if (buffer->off > buffer->sf_buf->off) {
! n = write(fd,
! buffer->buffer,
! buffer->off - buffer->sf_buf->off);
! } else {
! n = evbuffer_write(buffer->sf_buf, fd);
! }
! break;
! default:
! /* A sendfile buffer. */
! n = sendfile(fd, buffer->sf_fd, &buffer->sf_off, buffer->off);
! break;
! }
#else
n = send(fd, buffer->buffer, buffer->off, 0);
#endif
Here's a simple web server that uses sendfile:
/*
A libevent sendfile webserver. Opens a single file, stats it to get
its size, and serves it via a sendfile evbuffer.
Compile with
cc -levent sfhttpd.c -o sfhttpd
The run
sfhttpd <file>
where file is a large file.
Then point apachebench at it, e.g.
ab -n 10000 -c 1000 localhost:8080/
*/
#include <evhttp.h>
#include <fcntl.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <sys/types.h>
int fd;
off_t size;
static void
http_callback(struct evhttp_request *req, void *ign)
{
/* To turn an existing, empty evbuffer into a sendfile buffer,
assign it a sendfile fd (sf_fd) and set length and offset
of the data. */
req->output_buffer->sf_fd = fd;
req->output_buffer->sf_off = 0;
req->output_buffer->off = size;
evhttp_send_reply(req, HTTP_OK, "OK", NULL);
}
int
main(int argc, char **argv)
{
char *file = argv[1];
fd = open(file, O_RDONLY);
struct stat stat;
fstat(fd, &stat);
size = stat.st_size;
event_init();
struct evhttp *httpd = evhttp_new(NULL);
evhttp_bind_socket(httpd, "127.0.0.1", 8080);
evhttp_set_gencb(httpd, http_callback, NULL);
event_dispatch();
}
Whaddaya think?
Manuel Simoni
_______________________________________________
Libevent-users mailing list
Libevent-users@monkey.org
http://monkeymail.org/mailman/listinfo/libevent-users
