[Libevent-users] [PATCH] Add callback for raw read/write
olla,
I dont know what the process for submitting patches here,
but this is against version 1.4.9.
This makes it easier to integrate libraries that provide their own
read/write functions (such as openssl).
Example usage would be something like:
-
static
size_t myrcb(int fd, void *buf, size_t len, int flags, void *rcbarg)
{
fprintf(stderr, myrcb invoked\n, fd,len,flags);
blah;
bleh;
return specialized_read(fd, buf, len);
}
//somewhere in setup code ..
..
struct evbuffer rbuf;
..
...
evbuffer_setrcb(rbuf, myrcb, NULL);
..
...
//somewhere in callback
len = evbuffer_read(rbuf, fd, BUFSIZ);
-
Of course you could use bufferevents instead of direct calls
to evbuffer_read/write.
cheers,
jamal
diff --git a/buffer.c b/buffer.c
index e66080f..6032cad 100644
--- a/buffer.c
+++ b/buffer.c
@@ -341,16 +341,28 @@ evbuffer_drain(struct evbuffer *buf, size_t len)
}
+#define EVBUFFER_MAX_READ 4096
+size_t
+buffer_read(struct evbuffer *buf, int fd, int howmuch)
+{
+ /* We can append new data at this point */
+ u_char *p = buf-buffer + buf-off;
+
+ if (buf-rcb != NULL)
+ return buf-rcb(fd, p, howmuch, 0, buf-rcbarg);
+#ifndef WIN32
+ return read(fd, p, howmuch);
+#else
+ return recv(fd, p, howmuch, 0);
+#endif
+}
/*
* Reads data from a file descriptor into a buffer.
*/
-#define EVBUFFER_MAX_READ 4096
-
int
evbuffer_read(struct evbuffer *buf, int fd, int howmuch)
{
- u_char *p;
size_t oldoff = buf-off;
int n = EVBUFFER_MAX_READ;
@@ -383,14 +395,8 @@ evbuffer_read(struct evbuffer *buf, int fd, int howmuch)
if (evbuffer_expand(buf, howmuch) == -1)
return (-1);
- /* We can append new data at this point */
- p = buf-buffer + buf-off;
-#ifndef WIN32
- n = read(fd, p, howmuch);
-#else
- n = recv(fd, p, howmuch, 0);
-#endif
+ n = buffer_read(buf, fd, howmuch);
if (n == -1)
return (-1);
if (n == 0)
@@ -405,16 +411,23 @@ evbuffer_read(struct evbuffer *buf, int fd, int howmuch)
return (n);
}
-int
-evbuffer_write(struct evbuffer *buffer, int fd)
+ssize_t
+buffer_write(struct evbuffer *buf, int fd)
{
- int n;
-
+ if (buf-scb != NULL)
+ return buf-scb(fd, buf-buffer, buf-off, 0,buf-scbarg);
#ifndef WIN32
- n = write(fd, buffer-buffer, buffer-off);
+ return write(fd, buf-buffer, buf-off);
#else
- n = send(fd, buffer-buffer, buffer-off, 0);
+ return send(fd, buf-buffer, buf-off, 0);
#endif
+}
+
+int
+evbuffer_write(struct evbuffer *buffer, int fd)
+{
+ int n = buffer_write(buffer, fd);
+
if (n == -1)
return (-1);
if (n == 0)
@@ -449,3 +462,18 @@ void evbuffer_setcb(struct evbuffer *buffer,
buffer-cb = cb;
buffer-cbarg = cbarg;
}
+
+void evbuffer_setrcb(struct evbuffer *buffer,
+ssize_t (*cb)(int, void *, size_t, int, void *),
+void *cbarg)
+{
+ buffer-rcb = cb;
+ buffer-rcbarg = cbarg;
+}
+void evbuffer_setscb(struct evbuffer *buffer,
+ssize_t (*cb)(int, const void *, size_t, int, void *),
+void *cbarg)
+{
+ buffer-scb = cb;
+ buffer-scbarg = cbarg;
+}
diff --git a/event.h b/event.h
index b6a8144..e2e01f2 100644
--- a/event.h
+++ b/event.h
@@ -729,6 +729,10 @@ struct evbuffer {
void (*cb)(struct evbuffer *, size_t, size_t, void *);
void *cbarg;
+ ssize_t (*rcb)(int s, void *buf, size_t len, int flags, void *rcbarg);
+ void *rcbarg;
+ ssize_t (*scb)(int s, const void *buf, size_t len, int flags, void *scbarg);
+ void *scbarg;
};
/* Just for error reporting - use other constants otherwise */
@@ -1118,6 +1122,30 @@ u_char *evbuffer_find(struct evbuffer *, const u_char *, size_t);
*/
void evbuffer_setcb(struct evbuffer *, void (*)(struct evbuffer *, size_t, size_t, void *), void *);
+/**
+ Set a callback to invoke for reading from a file descriptor.
+
+ This is useful if you want to use your own read function instead
+ of the standard system call.
+
+ @param buffer the evbuffer to read into
+ @param cb the callback function to invoke for reading
+ @param cbarg an argument to be provided to the callback function
+ */
+void evbuffer_setrcb(struct evbuffer *, ssize_t (*)(int, void *, size_t, int, void *), void *);
+
+/**
+ Set a callback to invoke when the evbuffer is modified.
+
+ This is useful if you want to use your own write function instead
+ of the standard system call.
+
+ @param buffer the evbuffer to be written
+ @param cb the callback function to invoke when writting the evbuffer
+ @param cbarg an argument to be provided to the callback function
+ */
+void evbuffer_setscb(struct evbuffer *, ssize_t (*)(int, const void *, size_t, int, void *), void *);
+
/*
* Marshaling tagged data - We assume that all tags are inserted in their
* numeric order - so that unknown tags will always be higher than the
___
Libevent-users mailing list
Libevent-users@monkey.org
http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] [PATCH] Add callback for raw read/write
On Thu, Feb 19, 2009 at 5:09 AM, jamal h...@cyberus.ca wrote: This makes it easier to integrate libraries that provide their own read/write functions (such as openssl). I haven't looked at the patch, but I'm a little skeptical of a one-size-fits-all approach to integrating things like openssl. As the openssl faq says, you can't just call its read and write functions and expect them to work like regular ones: http://www.openssl.org/support/faq.html#PROG10 Do you have an example app that shows your code working with openssl? - Dan p.s. FWIW, here's how chromium uses libevent with nss (sorry, it's a little convoluted): http://src.chromium.org/viewvc/chrome/trunk/src/net/base/nss_memio.c http://src.chromium.org/viewvc/chrome/trunk/src/net/base/nss_memio.h http://src.chromium.org/viewvc/chrome/trunk/src/net/base/ssl_client_socket_nss.cc http://src.chromium.org/viewvc/chrome/trunk/src/net/base/ssl_client_socket_nss.h http://src.chromium.org/viewvc/chrome/trunk/src/base/message_pump_libevent.cc http://src.chromium.org/viewvc/chrome/trunk/src/base/message_pump_libevent.h I do have a standalone demo I could dig out if people are interestd. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] [PATCH] Add callback for raw read/write
On Thu, 2009-02-19 at 05:35 -0800, Dan Kegel wrote: I haven't looked at the patch, sufficient to look at the simple example usage shown in the email but I'm a little skeptical of a one-size-fits-all approach to integrating things like openssl. As the openssl faq says, you can't just call its read and write functions and expect them to work like regular ones: http://www.openssl.org/support/faq.html#PROG10 For pedagogical reasons openssl maybe a bad example to reference. libcurl and a few others which couple IO event management with their processing maybe a better example. openssl TLS (i suspect DTLS would be kinder) buffers at least a complete packet received for processing. I have never read the FAQ but the man page is atrocious at best, and I was bitten by what the FAQ describes - it was easy to see with strace what SSL read was doing. So when you do your wrappers that invoke SSL_read/write, you need to do a lot more. Example for read, pick your poison: - block until all is read - ask it to read you upto 16K buffer - keep invoking SSL_read in your wrapper until it tells you its done - set something in that invokes some IO scheduling you maintain based on what the SSL_read/error returns - etc In any case, I dont want to distract and justify the merits of the patch based on how it solves the tls issue. Do you have an example app that shows your code working with openssl? Not publicly; if there was some use to demonstrate usage using openssl, I could rip off some of the code and demonstrate how it can be done. But it will take me time to make it usable for other people. p.s. FWIW, here's how chromium uses libevent with nss (sorry, it's a little convoluted): I suspect it suffers from the same buffering issue as openssl. The reason i picked openssl is because it can do DTLS and there are experimental patches for SCTP. Ive never looked at NSS, but i glanced briefly at GNUTLS and it didnt meet my requirements. cheers, jamal ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Please check svn trunk before you write feature patches [was Re: [Libevent-users] [PATCH] Add callback for raw read/write]
On Thu, Feb 19, 2009 at 08:09:18AM -0500, jamal wrote: olla, I dont know what the process for submitting patches here, but this is against version 1.4.9. This makes it easier to integrate libraries that provide their own read/write functions (such as openssl). Hi, Jamal! Thanks for the patch, but the Libevent 2 dev series (currently calling itself 1.4.99) handles evbuffers and bufferevents significantly differently. (Old code still works, but the implementation is more efficient.) We're trying to keep the 1.4.x series as stable as possible, and put new features into 2.0. This reminder goes for everybody writing patches: if you have a feature patch (not a bugfix) that you want to go into the next version of Libevent, please make sure your patch is against the trunk of the subversion repository, not against the latest stable release. (Instructions for getting Libevent from subversion are here: http://sourceforge.net/svn/?group_id=50884 The URL you want to check out is https://levent.svn.sourceforge.net/svnroot/levent/trunk/libevent ) (As for the actual patch, it looks a lot more like the kind of problem we'd try to solve in 2.0 with a specialized bufferevent implementation.) yrs, -- Nick ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: Please check svn trunk before you write feature patches [was Re: [Libevent-users] [PATCH] Add callback for raw read/write]
Hi Nick, On Thu, 2009-02-19 at 10:33 -0500, Nick Mathewson wrote: Thanks for the patch, but the Libevent 2 dev series (currently calling itself 1.4.99) handles evbuffers and bufferevents significantly differently. (Old code still works, but the implementation is more efficient.) We're trying to keep the 1.4.x series as stable as possible, and put new features into 2.0. Ok (As for the actual patch, it looks a lot more like the kind of problem we'd try to solve in 2.0 with a specialized bufferevent implementation.) From a cursory glance of 2.0, the issue I am trying to resolve is still there - so i will migrate to 2.0 and then send a patch after playing with it for a bit. cheers, jamal ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
[Libevent-users] dumb question
I hope i come out sound patronizing or putting down the good work done in libevent. Is libevent trying to be too many things? I love the all-things-IO libevent provides; i guess buffer events are a natural evolution path - but why all the DNS or HTTP stuff? whats next? Is the end goal to become the one stop shop for all protocols (like python twisted)? I know i dont have to use or compile all the goodies, but would it not make more sense to keep the protocol processing engines as a separate library that uses libevent (or for that matter whatever the competition is)? cheers, jamal ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
[Libevent-users] Re: dumb question
On Thu, 2009-02-19 at 11:13 -0500, jamal wrote: I hope i come out sound patronizing or putting down the good work done in libevent. Sorry: I meant quiet the opposite ;- i.e hope i dont come out sounding patronizing. cheers, jamal ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] dumb question
On Thu, Feb 19, 2009, jamal wrote: I hope i come out sound patronizing or putting down the good work done in libevent. Is libevent trying to be too many things? I love the all-things-IO libevent provides; i guess buffer events are a natural evolution path - but why all the DNS or HTTP stuff? whats next? Is the end goal to become the one stop shop for all protocols (like python twisted)? I know i dont have to use or compile all the goodies, but would it not make more sense to keep the protocol processing engines as a separate library that uses libevent (or for that matter whatever the competition is)? I brought this up with Nick a couple weeks ago when we bumped into each other. I raised the possibility of breaking out the non-event code into separate libraries with enforced API boundaries. We were talking about various directions 2.0 can go in (in the context of doing sensible async IO that will scale under both windows and unix, given their differences of opinion in APIs :) but Nick's design goals differ slightly from my ideals. Adrian ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] dumb question
jamal wrote: Is libevent trying to be too many things? I love the all-things-IO libevent provides; i guess buffer events are a natural evolution path - but why all the DNS or HTTP stuff? The DNS and HTTP stuff is there so people don't have to constantly re-invent those tasks themselves. How often do you connect to a remote host without doing DNS lookups? From another perspective, another completely different library for asynchronous I/O - Boost.Asio - also provides asynchronous resolver functions, because the author of that library had pretty much exactly the same conclusion. For HTTP... well, the doxygen on the subject seems pretty self-explanatory: As libevent is a library for dealing with event notification and most interesting applications are networked today, I have often found the need to write HTTP code. The following prototypes and definitions provide an application with a minimal interface for making HTTP requests and for creating a very simple HTTP server. The server software I've written that *didn't* include an embedded web server for management was being moved in that direction to meet the needs of the operational staff; it's just part and parcel of servers these days. Right now I'm using Boost.Asio instead of libevent (probably better if you're working in C++, but not an option in C), but because of that I'm having to do work that libevent provides out of the box. whats next? What other protocols or services are commonly required by virtually every client, or virtually every server? Is the end goal to become the one stop shop for all protocols (like python twisted)? I think the end goal is to be a useful library for writing clients and servers that want to do I/O asynchronously. I know i dont have to use or compile all the goodies, but would it not make more sense to keep the protocol processing engines as a separate library that uses libevent (or for that matter whatever the competition is)? It could make sense, but what is the benefit to the separation? How many people using libevent will never use evdns or evhttp? -- Matthew Weigel hacker unique idempot . ent ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] dumb question
Hello all, if with async io is meant doing native async disc / file io, then there already is code that integrates with libevent-2.0. Recently, Valery Kholodkov and myself did some work in this direction. The latest version you can find in Valery's git tree right here: http://github.com/vkholodkov/libevent/tree/valery-master . What I can say is, that the linux native AIO was stable with several long-running tests ( 300mb/s for longer than 1 day) even in multi-threaded environments. I am not up to date with the latest libevent developments @ trunk, so please just ignore this email if it might prove redundant! Martin Adrian Chadd wrote: On Thu, Feb 19, 2009, jamal wrote: I hope i come out sound patronizing or putting down the good work done in libevent. Is libevent trying to be too many things? I love the all-things-IO libevent provides; i guess buffer events are a natural evolution path - but why all the DNS or HTTP stuff? whats next? Is the end goal to become the one stop shop for all protocols (like python twisted)? I know i dont have to use or compile all the goodies, but would it not make more sense to keep the protocol processing engines as a separate library that uses libevent (or for that matter whatever the competition is)? I brought this up with Nick a couple weeks ago when we bumped into each other. I raised the possibility of breaking out the non-event code into separate libraries with enforced API boundaries. We were talking about various directions 2.0 can go in (in the context of doing sensible async IO that will scale under both windows and unix, given their differences of opinion in APIs :) but Nick's design goals differ slightly from my ideals. Adrian ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] dumb question
On Thu, Feb 19, 2009, Nick Mathewson wrote: If I understand correctly, Adrian, you also think that bufferevent-ish stuff should have API-separation from the event-ish stuff (which I believe in) and that it should go into a separate library from libevent_core (which I don't agree with, but I don't feel too strongly about). I think its fine if they're in the same project, but I don't think they should be sharing library space or header files. That way you can be 100% sure that stuff doesn't depend on other stuff unexpectedly - you'll get compile / link errors. So you'd have libevent_core, libevent_dns, libevent_http, etc, etc. The remaining question here is whether (and to what extent) to split non-libevent-core stuff into a separate source package. I don't feel strongly here either; there are arguments for keeping it in one source package and arguments for splitting it. Personally, I think it's one of those bikeshed issues whose accessibility garners it attention beyond its actual impact. Probably we should revisit it after Libevent 2.0 is out; there is enough architectural stuff slated for 2.0 already IMO. I'm happy to wait for post-2.0 to discuss this stuff further. I don't have enough time to discuss it now. :) adrian ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
