On Sat, Nov 03, 2007 at 03:45:39PM -0700, William Ahern <[EMAIL PROTECTED]> wrote: > A good itch, indeed.
I am currently working on integrating all modules from libevent, so it becomes a full libevent replacement (and it already runs all of the testsuite that doesn't require access to internals). > > * there is full support for fork, you can continue to use the event loop > > in the parent and child (or just one of them), even with quirky backends > > such as epoll. > > Curious how you managed to do this. Are you checking the process PID on each > loop? I considered that, but I think its too slow (one also needs to be careful that watchers don't change e.g. epoll state until the getpid check is done), or at leats I think I don't want that speed hit, no matter what. Instead, I make it the users job to actually call me after a fork. I provide three functions: void ev_fork_prepare (void); void ev_fork_parent (void); void ev_fork_child (void); which you cna simply plug into pthread_atfork and it will work. The reason I don't myself is that I don't want to require pthreads just for that, but the perl interface for example does, so perl programs will be safe. I wrote "full support for fork" even though its not automatic because it can be done and is fully supported. With libevent, you can't free the event base in general (program crashes with an assertion has has been documented on this list a number of times). > > * there are two types of timers, based on real time differences and wall > > clock time (cron-like). timers can also be repeating and be reset at > > almost no cost (for idle timeouts used by many network servers). time > > jumps > > get detected reliably in both directions with or without a monotonic > > clock. > > But then they're not truly "real-time", no? Within the limits of technology, they are: - timers (based on monotonic time) will time out after "n" seconds (whatever was configured), even if the date resets in between (libevent can do that only for backward time jumps). - periodicals will simply be rescheduled, if a periodic timer is scheduled to fire "at" some point then it will not be affected by the time jump, it will still fire at that point (its more complicated with periodic timers scheduled to repeat, if you schedule a periodic timer to execute on every minute than libev will try to schedule it to occur when time() % 60 == 0, regardless of any time jumps. Of course, detecting and correcting this cnanot be done completely reliable with sub-second precision (there is no API in posix to do that), but with a monotonic clock, libev should manage quite fine to detect even very small time jumps caused by ntpd. (With no monotonic clock its a heuristic, obviously). > > * event watchers can be added and removed at any time (in libevent, > > removing events that are pending can lead to crashes). > > This is news to me. Can you give more detail, maybe with pointers to code? That is how I read the code in event.c on first glance. But in fact, it seems to be safe. I initially thought only removing the current watcher is safe. (I was in fact fooled by some bugs in the testsuite). Sorry for the confusion, I was too busy implementing all the other goodies, and right now I am busy implementing the remaining parts to get 100% event API support. > > * different types of events use different watchers, so you don't have > > to use an i/o event watcher for timeouts, and you can reset timers > > seperately from other types of watchers. Also, watchers are much smaller > > (even the libevent emulation watcher only has about 2/3 of the size of a > > libevent watcher). > > libevnet does this for I/O; timer is always set separately from read/write > events. (Point being, its using libevent.) Even libevent is somewhat fast if you don't combine timeouts and io watchers in the same struct event. But it is of course quite the waste. > > * I added idle watchers, pid watchers and hook watchers into the event loop, > > as is required for integration of other event-based libraries, without > > having to force the use of some construct around event_loop. > > Needing to do an operation on every loop is arguably very rare, and there's > not much burden in rolling your own. Its a quality issue. If you have a program that uses libevent and a library that needs to hook into it, it simply cannot be done. I happen to have many such cases. It basiclaly happens as soon as you use libevent as some part of some big program (such as in form of a perl module :), where many components might want to hook into the event loop. With that functionality in place, you can do it. Without it, you simply fail. It doesn't matter much, as libev is still faster than libevent even with all those watcher types. > PID watchers, likewise... how many spots in the code independently > manage processes (as opposed to one unit which can just catch > SIGCHLD). You could say the same about signals and be right just as well, they are not as important as io and timers. But a high-quality implementation should support these, as those are events, too, and, like select/poll etc. you can't share them within a single process, there must be a single event scheduler for that. So yes, they are not extremely important, but when you need that fucntionality and it isn't there, you simply fail. Again, think about a library hooking into this mechanism. As a practical example, again perl has an AnyEvent module. You can use i/o, timer and signal/child events portably with AnyEvent, and when using it, your software module will magically work under Gtk, libevent, Tk etc. > Also, curious how/if you've considered Win32 environments. I consider them a goal, albeit a low-priority one (until I find need for it myself). If libev code gets integrated into libevent it of course is a must to be portable to this unfortunately important platform. Likewise if somebody would need it and I could make it happen. Fortunately, libevent sets a good precedent and has lots of code. For example, I just wrote a kqueue interface, by looking at the libevent one, which has very important portability hints. Although libev backends are much simpler than libevent backends, I guess its even simpler to compile/debug something that has already been written, so I am confident kqueue will be supported soon. I will also provide generic backends for other platforms when people need them (and can provide me with documentation). win32 is certainly very highon the list. > > * the backends use a much simpler design. unlike in libevent, the code to > > handle events is not duplicated for each backend, backends deal only > > with file descriptor events and a single timeout value, everything else > > is handled by the core, which also optimises state changes (the epoll > > backend is 100 lines in libev, as opposed to >350 lines in libevent, > > without suffering from its limitations). > > libevnet optimizes state changes. Logically every I/O request is single-shot > (which is more forgiving to user code), but it actually sets EV_PERSIST and > delays libevent bookkeeping until the [libevnet bufio] callback returns. If > the user code submits another I/O op from its callback (highly likely) then > the event is left unchanged. It's still re-entrant safe because it can > detect further activity up the call chain using some stack message passing > bits (instead of reference counting because I also use mem pools, but I > digress). Again, point being this can be done using libevent as-is. Yes, in a very awkward way, with high resource wastage. My point being is that if I have a high-performance event loop that claims to support all these fine things than it should really not force it on the user to work around its performance problems. > > As for compatibility, the actual libev api is very different to the > > libevent API (although the design is similar), but there is a emulation > > layer with a corresponding event.h file that supports the event library > > (but no evbuffer, evnds, evhttp etc.). > > Well... if you can persuade me of the utility then this Christmas I might > want to investigate writing an evdns-like component. Well, there is no harm in using the libevent api. In fact, you probably should. The evdns code is not something I really look forward to running it (same with evhttp), but it exists, and is libevent compatible. If you stay within the libevent api, you get the best of all worlds, of course :) This can be seen clearly by the script I use to programmatically import the libevent code into libev: http://data.plan9.de/import_libevent *All* the code changes done there are actually either bugfixes or disabling parts of the testsuite that work around stuff or rely on internal structure members not exposed through the API. And you get almost all the advantages of libev even through the libevent API, and even if you don't separate timers from I/O. And it uses less memory (watcher size ~150 vs. ~120 bytes) and is faster too. And you can always go back to libevent if problems come up! :) If you want to have a look, here is the libev API header file (which I think would make a fine API for libevent, too): http://cvs.schmorp.de/libev/ev.h (all the macro magic exists so one can build libev with single-event-loop and multiple-event-loop interface. It also deals with including the .c files directly in your package, where you can then configure custom fields for your app to be included into all watchers). Here is the perl documentation. It has a slightly diffeernt interface, but it explains some of the rationale behind the timers, watcher lifetime and the other watchers: http://cvs.schmorp.de/EV/README Given that all this is only a few days old, be nice to me :) I expect to have some basiclaly workign libev release to be ready in a few days or so, though. Right now its sitll kind of a battlefield. > See the "lookup" component of libevnet. There are lots of nice things > I need in a DNS resolver that evdns and others are incapable of > handling. And I've also written more HTTP, RTSP, and SOCKS5 parsers than > I can remember. I will. As a minimum, the interface should be able to let me query a few things comfortably and any other type(s) of records with some less ease, and I will buy it immediately :) Try getting it into libevent so I cna pick it up and everybody else can take advantage of it :) > If you ask me, it would prove more fortuitous to re-write the DNS and HTTP > components then to replace libevent. Reason being because it would be hard > to substantively improve on DNS/HTTP without altering the API, whereas > clearly its feasible to improve libevent under the hood without altering the > existing API, and then building your new features on top of this. That's > sort of what I did with libevnet, by adding buffered I/O, DNS, and thread > management API atop libevnet. Yes, I see this now. I originally expected those components to be, well, not that full-features and correct (w.r.t. the protocol), but basically well-tested and working. The problem is that I am prepared to maintainer my event library, but I would really like to have some async dns, too, and the code exists, which is a good argument for using it (it *does* work, after all :) So for me, using all that code brings me a lot of features quickly. I am not conerned about bloat, as one can still include the full libev core without anythign else very easily: #define EV_STANDALONE 1 #define EV_MULTIPLICITY 0 #define EV_USE_EPOLL 1 #include "libev/ev.h" #include "libev/ev.c" > > So, would there be an interest in replacing the "core" event part of > > libevent with the libev code? If yes, there are a number of issues to > > solve, and here is how I would solve them: > > Win32 support is important to me, unfortuantely. As it likely is to others. > libevent has a very, very large installed base of users. Yes, another reason to make a high-quality emulation of its API. Even if it means relying on some bugs. Ok, you convinced me, I will look at how libevent handles win32 and likely adopt its code. It should be realtively cheap, as I already reuse the libevent configure stuff, too and have all the required files. It of course also helps that libev has less confifgure requirements. > > * libev only supports select and epoll. Adding poll would be trivial for me, > > I always thought it would be easier to just create kqueue wrappers around > epoll, poll, select, et al, and then build a library on that. Once you start > adding things like PID events, etc, its at least worth some thought. poll has been added already, btw. And I am not sure what you mean, the epoll backend alone is very simple, as are select and poll (all are aorund 100 bytes), while the kqueue backend is 160 lines already. It seems wrapping around kqueue would only add overhead (conceptually mostly in that you have to program more and more complicated stuff). Here is the epoll backend to illustrate how simple it really is: http://cvs.schmorp.de/libev/ev_epoll.c kqueue in itself doesn't add anything we can already do, too, as libev (just like libevent) has to provide stuff like signals and chld notifications portably, and having just one version of the code is betetr for reducing bugs and maintainance burden. Reminds me, libevent seems to forget about signals in one event loop when another one is running, I need to fix this in libev too, now that I have multiple event loops :/ > So you (1) want to support multiple destination events for the same source > but (2) let multiple threads signal discrete events? That sounds like an > invitation for even more trouble. Now you've got mutexes littered all over > the place. That's my version of a nightmare. And if you group discrete > events into the same thread, then you haven't solved the CPU workload > problem. > > I too prefer a single event base. The obvious problem being when you > actually have CPU intensive work, say an MPEG streaming service--lots of > I/O--which also needs to do intermedate AV processing--lots of CPU. Well, I usualyl solve this by havign a thread pool to do the work and integrating that one into my event loop. Works with any event loop, too, if you use a pipe and then use it sparingly. In any case, this entry is outdated, libev in CVS supports multiple event bases, in a fully compile-time optional way, so one can chose wether to have multiple event bases or have everything as static global variables. And the API is more obvious (for example, there is no event_set that temporairly re-assigns your active event to some other event base, as http.c currently does), but basically all functions that need it have as first argument the event base they need to work on. > Though, I think we've already had this debate... so... I'll just shut-up ;) I'll support it, I'll support it. I admit some of my views are policy and not requirement, and I admit I just didn't want to litter my API with an event_base, but I found a clean way to support both, so I will provide both. Thanks a lot for your input! -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / [EMAIL PROTECTED] -=====/_/_//_/\_,_/ /_/\_\ _______________________________________________ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
