I think I found a workable solution that leaves the user with total control if desired but otherwise achieves a reasonable behavior, too.
Path attached.
Thanks for finding this,
Niels
On 2/11/06, Niels Provos <[EMAIL PROTECTED]> wrote:
> Wow. What a strange circumstance. That must have been somewhat
> puzzling to debug. Your fix seems fine although - I think - it might
> not be sufficient. I don't think that anybody (but me and maybe a
> couple who read the source code) even know about the high-water mark
> feature.
>
> Perhaps there should be a limit by how much a buffer can be increased
> at any given time?
> Thank you very much for figuring this out,
>
> Niels.
>
> Cc'ing libevent users for more feedback.
>
> On 2/4/06, Christopher Maxwell <[EMAIL PROTECTED]> wrote:
> > It has taken quite a strange set of circumstances to do this, but I have
> > discovered a new way to exhaust system memory when trying to read from a
> > bufferevent.
> >
> > After externally copying a large file to another filesystem, such that a
> > very large portion of the file remains in the system buffer cache (UBC?),
> > the automatic calculation by bufferevent_readcb() and evbuffer_read()
> > notices the absurdly large number of bytes immediately available to read and
> > attempts to resize the evbuffer. This all happens before any callbacks can
> > be run to drain the evbuffer. In my case, a file of ~100MB appeared all at
> > once, was packaged into another 100MB outbound buffer, then promptly
> > exploded.
> >
> > I could not find any knobs to adjust as watermarks are not checked until
> > long after the buffer read. Below is a patch to bufferevent_readcb() that
> > constrains the read size to the high watermark, if set. The plan is to
> > give the application a fighting chance to drain the buffer, or at least
> > disable it for a time if it cannot.
> >
> > cheers
> > --chris
> > [EMAIL PROTECTED]
> >
> > --- evbuffer.c.orig Sun Feb 5 00:21:06 2006
> > +++ evbuffer.c Sun Feb 5 00:20:45 2006
> > @@ -92,13 +92,17 @@
> > int res = 0;
> > short what = EVBUFFER_READ;
> > size_t len;
> > + int howmuch = -1;
> >
> > if (event == EV_TIMEOUT) {
> > what |= EVBUFFER_TIMEOUT;
> > goto error;
> > }
> >
> > - res = evbuffer_read(bufev->input, fd, -1);
> > + if (bufev->wm_read.high != 0)
> > + howmuch = bufev->wm_read.high;
> > +
> > + res = evbuffer_read(bufev->input, fd, howmuch);
> > if (res == -1) {
> > if (errno == EAGAIN || errno == EINTR)
> > goto reschedule;
> >
> >
>
libevent.diff
Description: Binary data
_______________________________________________ Libevent-users mailing list [email protected] http://monkey.org/mailman/listinfo/libevent-users
