[Libevent-users] Re: IOCP API
From: Toby Douglass <[EMAIL PROTECTED]>
Okay, so I've finished writing the IOCP code.
I now need to finish the test/example application.
When that's done, I'll upload. Might even be this weekend.
Prototypes
==
1. int socket_iocp_new( void **socket_iocp_state, size_t
read_buffer_size, void (*accept_callback_function)(SOCKET socket, struct
sockaddr *local_address, struct sockaddr *remote_address, void
*listen_socket_user_state, void **per_socket_user_state), void
(*recv_callback_function)(SOCKET socket, void *read_buffer, DWORD
byte_count, void *per_socket_user_state), void
(*send_callback_function)( SOCKET socket, void *per_socket_user_state,
void *per_write_user_state ), void (*error_callback_function)(SOCKET
socket, void *per_socket_user_state) );
[...]
Comments and thoughts, please.
It looks to me like all your callbacks are asynchronous.
Is that the case? If so you're probably working off the standard
iocp example code which spawns num_cpus/2 or num_cpus*2 or
f(num_cpus) threads, however that's the sort of choice that
I think should be left to user, exposing instead an interface
int iocp_wait(long timeout) which calls the callbacks. This
lets the user choose the "asynchronicity" of their app,
i.e. they can write singlethreaded mutex-less iocp code.
As an interface socket_iocp_new is a tiny bit "busy" as it
wraps almost every function that iocps understand. A bit of
factoring might make it more friendly. A control-block style
interface could also help as it cuts down the argument list
and makes the "per socket user state" implicit (it actually
makes it per iocp operation user state). It also gives you
two way communication (user -> iocp library -> user in callback)
typedef struct {
SOCKET s; /* <- */
char* buf;/* <-, -> */
longnbytes; /* <-, -> */
void (*iocp_read_callback)(iocp_state* st, iocp_read* control_block); /*
<- */
} iocp_read;
The user can piggy back per operation data like so:
typedef struct {
iocp_read control_block;
/* extra stuff here */
}my_iocp_read;
void
my_iocp_read_callback)(iocp_state* st, iocp_read* control_block) {
my_iocp_read* my_cb = (my_iocp_read*)control_block;
/* respond, requeue, whatever */
}
RF
--
felix: lightweight threads, HOFs and game scripting.
svn co https://svn.sourceforge.net/svnroot/felix/felix/trunk felix
___
Libevent-users mailing list
Libevent-users@monkey.org
http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] Re: IOCP writes
From: "Toby Douglass" <[EMAIL PROTECTED]> I wrote: > On a related note, did someone say that the event handle in the overlapped > structure used by the IO functions (ReadFile, etc) is not actually used? > > If that's actually untouched, I could use it for my own purposes. The > alternative is probably a malloc() per write operation :-/ As it is, it's easy to support serial writes with IOCP, it's supporting multiple concurrent writes which is awkward for a convenient-to-use API over IOCP - and this I would think is absolutely required for some uses of UDP sockets. I haven't tried any UDP async socket stuff, not with IOCPs nor anything. Tell us how it goes! Still, I think I'm missing the point of multiple concurrent writes - wouldn't the output be undefined? -- felix: lightweight threads, HOFs and game scripting. svn co https://svn.sourceforge.net/svnroot/felix/felix/trunk felix ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] Re: IOCP writes
From: "Toby Douglass" <[EMAIL PROTECTED]> >> I've realised, though, that if you issue a write on a socket which >> currently has an outstanding reading, when the IOCP completes, you don't >> immediately know which operation has completed. > > Obviously I used the completion key. Actually, that's wrong. On a related note, did someone say that the event handle in the overlapped structure used by the IO functions (ReadFile, etc) is not actually used? If that's actually untouched, I could use it for my own purposes. The alternative is probably a malloc() per write operation :-/ You shouldn't have to touch that stuff. What are you trying to do? Issuing simultaneous asynchronous reads and writes on socket is a tiny bit rare and special. Are you sure that's what you want to do? If so, great - io completion queues can handle it. Bu-ut if libevent was modelled on the unix select-style interfaces, most of which don't easily support this kind of thing, how is this situation cropping up, assuming that this is just an IOCP port of libevent? The unixy case is trickier - I think only kqueues can handle two separate read/write readiness requests being queued up in separate invocations. With all the other interfaces (epoll, event ports, select) you can request read AND write readiness notification, but you have to do with the one system call (the select interface makes this obvious, the other less so). -- felix: lightweight threads, HOFs and game scripting. svn co https://svn.sourceforge.net/svnroot/felix/felix/trunk felix ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] Re: IOCP writes
From: "Toby Douglass" <[EMAIL PROTECTED]> > I'm adding support for IOCP writes. > > I've realised, though, that if you issue a write on a socket which > currently has an outstanding reading, when the IOCP completes, you don't > immediately know which operation has completed. Obviously I used the completion key. I didn't see it immediately because I'm already using that for something else, bah, hambug. I don't think you should be passing the same overlapped structure twice to WSARecv/Send. I'm pretty sure that the overlapped isn't "yours" until GetQueuedCompletionStatus says so. Try a unique overlapped structure for each overlappable call. -- felix: lightweight threads, HOFs and game scripting. svn co https://svn.sourceforge.net/svnroot/felix/felix/trunk felix ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] Re: IOCP writes
From: "Toby Douglass" <[EMAIL PROTECTED]> I'm adding support for IOCP writes. I've realised, though, that if you issue a write on a socket which currently has an outstanding reading, when the IOCP completes, you don't immediately know which operation has completed. You use two different OVERLAPPED structures. The pointers returned by GetQueuedCompletionStatus will be different. The obvious answer is to use select() to check the socket when an IOCP completes, but that's very awkward, because of the race conditions, for the other operation could complete in the time between the IOCP complete and select() and you have multiple threads calling GQCS concurrently. Locking would be required, which would be Bad. That's doesn't sound right..., IOCP type progs really shouldn't need select. RF -- felix: lightweight threads, HOFs and game scripting. svn co https://svn.sourceforge.net/svnroot/felix/felix/trunk felix ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] iocp status?
How's the IOCP support going? I don't see anything about it on http://monkey.org/~provos/libevent/ -- felix: lightweight threads, HOFs and game scripting. svn co https://svn.sourceforge.net/svnroot/felix/felix/trunk felix ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] Re: Win32 I/O completion ports code release
From: "Toby Douglass" <[EMAIL PROTECTED]> > Use AcceptEx. The docs sayeth; "The following functions can be used to start I/O operations that complete using completion ports." (And then the above list is given). This to me means that it is these and only these functions which can be used with IOCP, which in turn means having an overlapped structure doesn't automatically mean you can use that function with an IOCP. Have you used AcceptEx() with an IOCP and found it works? Yes! and Yes! ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] GetQueuedCompletionStatus return values question/problem
From: Toby Douglass <[EMAIL PROTECTED]> Kevin Sanders wrote: > On 11/1/06, Toby Douglass <[EMAIL PROTECTED]> wrote: >> Kevin Sanders wrote: >> > Call GetLastError(). >> >> I could be wrong, but I don't think that helps you, because you would >> have to know the full list of possible errors for both cases, so that >> you could be certain that you were correctly differentiating between >> them. >> >> I have no idea what errors either cases might return, nor any idea of >> where to look to find the full list of possible errors for both cases >> and indeed I suspect that information is not documented. > > In case 3, GetLastError will not be 0, in case 4, GetLastError will be 0. In cast 4, will GLE always be 0 - Yesse. even if the other end has for example reset rather than disconnected gracefully? No... don't think so. The problem we face here is that although we can empirically find the GLE values for our current platforms, the possible GLE values are not documented! It's a bit like that, however write a few simple socket programmes and you'll quickly learn the win32/IOCP lingua franca, there's about a zillion examples over on koders.com, in fact probably the hardest thing about IOCPs is remembering how to spell "queued". so we can't be certain we've got it right, that there are no unusual cases we simply haven't seen, or that it won't change in the future. I'm happy to with this now, however, since it seems basically reasonable, although I do wonder at GLE being 0 for both a reset and a graceful disconnect! Are you 100% sure that's the case? Thankyou for your help, Kevin. No worries, but I'm not Kevin. RF ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] GetQueuedCompletionStatus return values question/problem
From: Toby Douglass <[EMAIL PROTECTED]> This is for those of you who've done some IO completion port work. According to my reading of the docs, GetQueuedCompletionStatus has four return status'; --- 1. return value != 0 - all went well, we've just returned from a successful dequeuing Yes. 2. return value != 0 and lpOverlapped == NULL - all handles associated with the IO completion object have been closed and the IO completion object itself has been closed (time to quit) For an lpOverlapped == NULL case I recognise two err codes, WAIT_TIMEOUT (common, because you can use a demuxer in poll mode) and ERROR_OPERATION_ABORTED. This last one I've never seen... I think, or only when someone rudely closes the iocp from under the thread calling GetQueuedCompletionStatus, something that really shouldn't be done. 3. return value == 0 and lpOverlapped != NULL - GetQueuedCompletionStatus() was okay, but we dequeued a *failed* IO operation (note that lpNumberOfBytes is not set in this case) Really? What kind of io operation? WSASend/Receive? Connect/AcceptEx? 4. return value == ERROR_SUCCESS and lpOverlapped != NULL and *lpNumberOfBytes == 0 - the handle we've come back on is actually a socket, and the remote host has closed The handle could be other things, but ok... p.s. !GetQueuedCompletionStatus indicates success and then you always get your lpOverlapped, lpNumberOfBytes has different interpretations based on the op (mostly what zero indicates). --- Now, the problem is this: ERROR_SUCCESS is #defined as zero! So !return_code => success! In other words, I don't think it's possible to differentiate between case 3 and 4, because in case 3, *lpNumberOfBytes might be 0 just by chance. If you're talking about socket IO, don't worry - the iocp won't bother you in the case that nothing arrived or was sent, it's different to select in that it tells you about stuff that has actually happened. You can get zero bytes on a WSAReceive which indicates socket closure. Erm, what cases are you actually trying to detect, maybe we can draw up a table of return vals and numbytes non-zeroness and outcome for a given op. RF ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] questions about bufferevent
From: "Matt Pearson" <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> wrote: > It's pretty spectacular stuff, I had a fairly average xp box doing > 10k sockets > (through an external interface, no cheatingly going through localhost) Did you get to 10k+ sockets in your first effort, or did you have to tweak the OS config? Pretty much the first go, there was some weirdness with pre-binding an acceptex socket to a dud address, and then I added some special casing to the felix stdlib to map cat: FILE -> SOCKET to iocp TransmitFile (the test prog was written in flx, but don't worry, flx maps to c++ so there's an equivalent c++ prog of course). As for tweaking the OS (xp sp2), no, wouldn't know how, I'm not really a windows person, I did all the coding under cygwin+mingw. I have my own windowsable mac now, so I should dig out that old code, still it's a bugger to test when you don't have many computers. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] Re: sockets under Win32
> Doug, if you're looking for blistering socket performance you need to > drop all this and learn about I/O Completion Ports. I have clients > and servers which can drive more than 10k sockets with NO MORE threads > than you have CPUs as reported by GetSystemInfo. It's pretty spectacular stuff, I had a fairly average xp box doing > 10k sockets (through an external interface, no cheatingly going through localhost) without breaking a sweat and with minimal cpu usage. I was doing TransmitFile (also IOCPable) down a socket (like a webserver, minus the header and footer). The hard part was driving the test, I needed several machines, as a the single mac g5+ kqueues was in kernel panic terroritory at around 5000 sockets. To test throughput I tried chunky reads and writes across several interfaces, including a firewire card - that worked great, but the xp firewire/tcp impl occasionally wedged itself... Apparently the server versions of the windows have differently tuned versions of the IOCP type calls, but I've never had a copy to test. I guess they're just more "serverly". Boh. I started from the kegel c10k website (not to be confused with the pelvic floor exercises of the same name), however it doesn't have that much on IOCPs, being a tad unix-centric. The IOCP library code (but probably not the test itself..., should be added as some sort of regression test) is part of the felix tarball (felix.sf.net). Love, RF. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] libevent 1.2
From: William Ahern <[EMAIL PROTECTED]> On Wed, Oct 18, 2006 at 12:21:17AM -, Christopher Baus wrote: > I'm not recommending that this should be in libevent, but if you are > looking for something that abstracts out both IOCP and non-blocking > sockets, ASIO is a good place to start. > I wouldn't write a project in C++ if you paid me a million dollars. ;) Hmm, I suppose, but have you considered that that's a lot of money? RF ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] libevent 1.2
From: "Christopher Baus" <[EMAIL PROTECTED]> > Has there been much interest in an IOCP port for libevent? I would > certainly be interested. > I even started working on a rough port, but as mentioned IOCP is proactive > rather than reactive, > (ie you tell it how much to read/send and the event tells you that) and it > got rather awkward pretty quickly. In libdemux, the event library used by felix (felix.sf.net), I modelled the problem with "control blocks" (accept, connect, read, write, AcceptEx, ReadFileEx, TransmitFile, ecc) that contained their args, a start method and a finish callback. The control blocks interacted with the event source (select, epoll, iocp, evtports, kq, no signal stuff, setting sig masks is a pain and no aio_* interface yet cos it's not that compelling), associating themselves either via "start" (unix) or the constructor itself (win32/iocp). It works well, despite the above annoying difference which is probably a wrinkle inherent in the difference between "reactive" and "pro-active". There are threadsafe and non-threadsafe versions of the reactors (which I call demuxers, echoed by asio, so I guess I wasn't completely on drugs when I made that name up) and self pipe classes for those who like to exit multithreaded apps or get attention in a polite manner. I'm not happy with the win32 self pipe as it's not anonymous (anon pipes don't work with IOCPs, socketpair doesn't exist, AF_UNIX is unimplemented, inventing a pair by connecting to localhost raises the hackles of firewall software and so on), hence I need a way of uniquely creating named pipes or to find some flag that limits the scope of the named pipe. Unfortunately, I don't know much about windows. The license is FFAU (free for any use) so if anyone's interested, steal away! Of course I'd be interested in any bug fixes (including understanding the kq workaround) and grand demuxer unifying theorems. Suggestions on how to improve the timer events are also welcome (timer queues on win32 and timed cond var on unix, I think the interface I used in the latter renders it a bit naf). Be aware that the srcs have been made "literate" and so exist in a single file: lpsrc/flx_demux.pak and require one or two config style macros (existence of various event APIs, sock_addr_len type and other stuff that's easy to guess). The slickest implementation I've seen that works with *nix non-blocking I/O and Windows IOCP is boost ASIO http://asio.sourceforge.net/. Instead of modeling reactive behavior with IOCP, it mimics proactive behavior on *nix. It works out surprisingly well. Wow, that looks very complete and mature, even if the name does strike fear into the hearts of australians. There's even a self pipe class for unix... but I don't see a win32 version. From: William Ahern <[EMAIL PROTECTED]> I'm privy to my own [C] library which does this and more. Still, I don't think this necessarily belongs in libevent. libevent is popular because it does cross-platform event _polling_ very well, in a clean and simple interface. It polls? I thought the whole point of event interfaces was to avoid "polling" readiness, preferring instead "knowing", however unix style evt interfaces fall down a little in this respect, and offer only "suspecting", still, saves CPU. Integrating compound (higher-level) asynchronous actions like this is still a very fluid area. Take the recent DNS support in 1.2. It's probably going to follow the same tired path that c-ares, ADNS and UDNS have... slow and abortive patches for IPv6, SRV, CNAME, et al (granted UDNS has probably done the best out of all of them). Sure, once I'd covered basic socket readiness, file io and so on, I looked to DNS thinking there would be a similar readiness interface. There wasn't, instead existing various blocking implementations where you can either wait or spawn a thread to wait for you. That kind of wrapping seemed trivial and for similar reasons I didn't wrap aio_*, because the impls that I saw seem to be pthread+blocking read/write... In any case, most overlapped=IOCPable stuff in windows seems quite low-level, lower than DNS, which is basically a socket application rather than a fundamental io operation. libevent fills in its niche almost perfectly. If it starts to spill out all over the place things will only get messier; libevent originally made things cleaner. Maybe it's inevitable... I suppose that's how fluid areas become well trodden ground. Covering IOCPs isn't so hard nor polluting with the following interface: unix: demuxer d; control_block_x cb(args); // override "finish" to customise behaviour d.start(cb); d.wait(); win32/iocp: demuxer d; control_block_y cb(d, args); // override "finish" to customise behaviour d.start(); d.wait(); RF ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] libevent 1.2
On 10/17/06, Scott Lamb <[EMAIL PROTECTED]> wrote: On Oct 17, 2006, at 11:08 AM, Rhythmic Fistman wrote: > That's funny, there doesn't seem to be a self-pipe-trick > implementation in lib-evt... how do you get by without it? There is, although it uses a socketpair() instead of a pipe(). Ah, I was grepping "trick". It looks like it's used internally in the signal style implementations - so how does a multithreaded app get thread_dispatch to return when it's time to exit? ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] libevent 1.2
From: "Niels Provos" <[EMAIL PROTECTED]> libevent 1.2 was released yesterday. There are some major feature improvements: - Support for non-blocking DNS resolution - Support for lightweight HTTP clients and servers - Support for Sun's Event Ports Neato! Seems you've subsumed most OS event waiting interfaces. Does that include win32 IOCPs (io completion ports) yet? If not, they're definitely worth a look. However, they're not very "selecty", in that they scale well and they tell you when something has actually happened instead of when something might happen without blocking, maybe. I don't know how well that would fit in with libevt. P.S. What was the story with kqueue bug workaround for osx10.4 that appeared in libevt1.1? The "bug" detection code seems to consider the kq impl broken depending on how it reported the error of waiting for data on an invalid file descriptor... How did libevt come to depend on that kind of behaviour? Do you often wait for evts on invalid file descriptors? Is it some sort of convoluted method for waking the given select-style interface? That's funny, there doesn't seem to be a self-pipe-trick implementation in lib-evt... how do you get by without it? Anyway, I've wandered off the beaten track a bit. Toodles, RF ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
[Libevent-users] os x 10.4 kqueues bug
I notice that libevent now has a work around for a bug in kqueues on os x tiger (work around = don't use kqueues on that system). What exactly was the bug? I can't find any hard information about it on the net. The test for broken kqueues doesn't tell me much either, it checks the implementation's behaviour when monitoring an invalid fd (-1). What's it all about? ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
