Niels Provos wrote:
On Mon, May 12, 2008 at 3:33 AM, Cezary Rzewuski
<[EMAIL PROTECTED]> wrote:
I suspect there may be two reasons, but I don't know if this how libevent
works:
1) there is any priority in events processed in libevent, and buffer
deallocation remains in the tail of events queue
2) may be there is any HTTP header that causes libevent to postpone closing
connection (which crawler uses but wget not).
We would have to see your code to figure out what's going on. For
example, HTTP has persistent connections, i.e. connections that stay
open after a request has been handled. It would also be good to know
which version of libevent you are using.
Niels.
Actually my code is very similar to spybye. However, I don't need sites
to be cached or to wait for antivirus
scanning, so it's kind of simplified spybye. libevent functions are used
in exactly the same way as in spybye.
I use libevent1.4.3-stable.
After more extensive debugging I suspect, that, in case of heavy
workload, libevent just doesn't complete
all the requests that are scheduled.
I can see that my proxy receives a request from the crawler, sends the
request to a www server,
then receives response from the server. Then evhttp_send_reply function
is called to return
the reply to the crawler.
But then, strangely, function evhttp_write is never called with argument
of this particular connection. Consequently, evhttp_send_done function,
which is responsible for freeing connection and request structures, is
also never called.
This causes that evhttp_connection and evhttp_request structures are
never freed.
The situation I've described happens really randomly and only in case of
crawling (which means many requests).
When I use proxy with browser (and human is the one that interacts:)),
everything looks fine.
Cezary Rzewuski
_______________________________________________
Libevent-users mailing list
Libevent-users@monkey.org
http://monkeymail.org/mailman/listinfo/libevent-users
