Re: [Libevent-users] Must delete events before closing the socket handle
On Thu, Sep 17, 2009 at 07:51:58PM -0700, Gilad Benjamini wrote: I just resolved a similar issue today, where I needed to close a file descriptor AFTER deleting an event. My symptoms, though, were different. What underlying mechanism is your libevent using ? In my case it is epoll. With select or poll my scenario had no problems, in spite of the wrong order. Few people do it, but these--and other--errors would be easier to catch if people checked the return value of event_del(). I often use assert(0 == event_del()), because if event_del() is failing then in all likelihood I have a bug in my code (the alternative being a bug in libevent, or a kernel bug). 'Tis better to have QA screaming at you, or a few hundred upset customers, than to roll out a buggy app to tens-of-thousands of customers and watch tech support disintegrate as you struggle to locate the source of the bug, assuming you can even get a trace. (This is also why setting NDEBUG for the production build is usually counter-productive.) ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] patch to use different return value for event_base_loop()
On Wed, Aug 12, 2009 at 10:27:15AM -0700, Ka-Hing Cheung wrote: On Wed, 2009-08-12 at 02:50 -0700, q6Yr7e0o nIJDVMjC wrote: Hi, one general question, why do you use #defines instead of enums? Because the constants right above are also #defines, so I just follow the existing practice. Also, one bonus to defines is that you can do feature checks without being forced to use an autoconf-like system. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] Building with Visual Studio
On Thu, May 14, 2009 at 04:58:02PM -0700, Rush Manbert wrote: snip I have two questions: 1) Why doesn't libevent as released just build in Visual Studio if this was so easy to accomplish for me? Or have I missed something? I don't maintain libevent, but I'd guess any issues exist for the same reasons I've encountered in my own projects. _Patching_ to support a platform like Win32+VS is easy. _Maintaining_ the patch/port is quite a headache. Especially so regarding Visual Studio--as opposed to a simple MinSYS make(1) port--because the build system is probably never used by the lead maintainers. And in my short, punctuated experience w/ Visual Studio, the config system changes faster than even autotools' rate of churn (and I'm no fan of autotools). (Plus, it's common these days for environments to host a plethora of autotools version combinations, but because of the monolithic IDE paradigm of Visual Studio, this isn't feasible for typical Windows developers.) 2) I would be happy to provide a patch that adds this support. Are any of the developers interested in receiving it? Afraid of commitment? (Just kidding.) If libevent still doesn't build out-of-the-box, despite the endless submissions of patches to fix [apparently] broken builds over the years, I suspect the only tenable solution is for somebody to step up and either manually or automagically build from the tree on a regular basis (weekly?), or to fork a Windows branch which tracks the development branch, similar to how Portable OpenSSH tracks OpenBSD's OpenSSH. In my projects, I typically only attempt to support make(1). (GNU Make if I'm lazy, or the project is too big.) But this allows me to focus on using a mingw-gcc cross-compiler to verify builds and Wine to run regressions. The best VS patch to libevent would be one which somehow automates the process similarly. Alternatively, maintain a port uses CMake--or something similar--which might ultimately get pulled into mainline. But you'll need significant buy-in (i.e. have done a very polished job) from the leads, I think. Lastly, ditching Visual Studio support altogether--instead relying on MinSYS and make(1)--and simply maintaing a binary Visual Studio archive might work better. But unless it's kept up-to-date on a very regular basis, it'll be hard to placate a developer's need/greed for the latest, greatest, just-fished-out-of-the-water code (a near universal ailment ;) ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
[Libevent-users] Re: function delegates
I've considerably refined my earlier approach to libevent delegates / psuedo-lambda expressions. I've abstracted out function pointer delegation -- delegate() invoke() -- and then built the libevent wrapper -- event_delegate() -- around that. Other changes: - No more exponential pre-processor expansion (i.e. no more 2MB expansions). - invoke() can proxy return values (ignored from libevent wrapper). - Static detection of libevent's void(*)(int, short, void *) callback signature by event_delegate(); no libffi overhead. Useful for micro-optimizing without having to dramatically change code. - Supporting more than 7 arguments should be a snap to add. The C99 macro magic has been re-written to use recursive'ish techniques. GCC and libffi are, of course, still required. Code available at: http://25thandclement.com/~william/projects/delegate.c.html I've been using this in a recent project and am quite happy. Only headache is that it can be a pain to locate libffi headers, even if manually installed. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
[Libevent-users] function delegates
Attached is a crack at supporting type-independent callback handlers. I've gotten tired of stubbing out handlers everywhere. It uses libffi (distributed with GCC), and GCC's type introspection builtins. Strictly speaking, it'll never be portable; but in practice it should be quite portable, at least in mainstream unix and win32 environments. It also shouldn't run afoul of non-executable stack environments, but this is untested. event_delegate() works like event_set() + event_add(). It takes the (struct event_delegate) object, file descriptor, event set, timeout, callback function (generic: void (*)()), and 0 to 7 parameters to pass when invoking the callback. delegate_init() initializes the object, and delegate_reset() should be called at the end of the lifetime of the context; e.g. before control of fd is passed away to, say, close(2). These requirements are just an artifact of my implementation, not intrinsic to the concept. TODO: Add magic parameter constants, which will be replaced on invocation with a dynamic value. This would provide, for example, the ability to wrap existing callbacks which have the (int), (short), (void *) signature (i.e. where (short)events is passed as a parameter, the value for which is unknown when calling event_delegate()). I'm sure there are real, nifty uses for such a feature. Ultimately, this gets the C developer quite close to something like closures, which are so incredibly handy with event-oriented code in other languages. /* == * delegate.c - Polymorphic libevent callbacks. * -- * Copyright (c) 2009 William Ahern * * Permission is hereby granted, free of charge, to any person obtaining a * copy of this software and associated documentation files (the * Software), to deal in the Software without restriction, including * without limitation the rights to use, copy, modify, merge, publish, * distribute, sublicense, and/or sell copies of the Software, and to permit * persons to whom the Software is furnished to do so, subject to the * following conditions: * * The above copyright notice and this permission notice shall be included * in all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN * NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, * DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR * OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE * USE OR OTHER DEALINGS IN THE SOFTWARE. * -- * * CAVEATS: * * o Only pass integral types as callback parameters, and not compound *types like structures, unions or arrays. Pointers to compound *types are fine. Passing arrays directly should behave like passing *a pointer to the array, but caveat emptor. * * o Depends on pointer equivalence; or rather, that void pointers have *the same underlying representation as any other pointer type. * * o Only works with GCC, or compilers which can impersonate well GCC. * * o Uses libffi, which is distributed with GCC. * * o Requires support for C99 compound literals; i.e. -std=gnu99 or *-std=c99 (untested). * * o There shouldn't be side effects such as parameters being evaluated *more than once, but caveat emptor. * * o Maximum of 7 callback parameters; minimum of 0. * * o event_delegate() lazily uninstalls the underlying libevent *handler, since this is my preferred pattern with libevent. This *has several impliciations: * * + EV_PERSIST is implicit, however unless you re-issue *event_delegate(), the handler will be removed *automatically after a callback. You can call *event_delegate() as many times as you want, however, w/o *side effect. * * + delegate_reset() or delegate_del() MUST ALWAYS BE CALLED *BEFORE the underlying object memory is deallocted AND *BEFORE the file descriptor is closed or ownership passed; *otherwise, delegate_fwd() (internal function) will *dereference and potentially corrupt invalid memory and/or *spuriously remove an epoll/kqueue association. * * + Depending on your platform and version of libevent, you *must explicitly call delegate_reset() or delegate_del() *before any other code can re-register the underlying *descriptor itself. * *In practical terms, this merely means you
Re: [Libevent-users] http: libevent vs many threads
On Sat, Apr 19, 2008 at 12:23:16AM +0200, Springande Ulv wrote: On 18. april. 2008, at 22.36, Cezary Rzewuski wrote: our best bet is to use a single thread using libevent, or go totally multi-threaded without libevent. In 90% of the circumstances one of those options (though not both) Why not both?! Using both threads _and_ libevent is not only possible both often feasible. Use libevent in a i/o bound thread and a thread pool for CPU bound operations. In a network application the network is the bottleneck and a few context switches is diminutive in this context. Because _unless_ you can justify the additional complexity, why bother? In some sense--like code complexity--using both an event-oriented design *and* threads is the worst of both worlds. Your processing logic is turned inside-out (state machines, etc, can be confusing to some people), plus you have mutexes and barriers and all that crap littered throughout your code. I've used libevent with threads. I've also used libevent with multi-process configurations. But, usually I stick with one or the other. Premature optimization is the root of all evil ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] http: libevent vs many threads
On Sat, Apr 19, 2008 at 02:49:20AM +0200, Springande Ulv wrote: On 19. april. 2008, at 01.57, William Ahern wrote: In some sense--like code complexity--using both an event-oriented design *and* threads is the worst of both worlds. Your processing logic is turned inside-out (state machines, etc, can be confusing to some people), plus you have mutexes and barriers and all that crap littered throughout your code. He he, the crack you just heard is the thin ice you walked out on. I must be deaf. Premature optimization is the root of all evil Oh that is an old misunderstood meme. What Hoare actually said was We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. He was talking about micro-optimizing code. Not exactly what I would call thinking about a scalable program design, which one should think about from the start and at all time, unless you work in the Microsoft Office team. I really couldn't care less what Hoare or anybody else said. I'm not appealing to authority. It's an apposite phrase. If you fancy all your first or even second iterations of your projects running a gigantic data center, *and* you actually meet your targets out the gate, then good for you. (Normal people would write a proof-of-concept first, and maybe second, and third.) But, in my experience it's best to aim for correctness first. If you must, you can fix most any well designed single process event-oriented daemon to scale [more] by tweaking it to use multiple processes, or multiple servers. The same more-or-less goes for threaded servers. But you usually don't need to, because as long as you're not needlessly copying data or doing other wasteful things, then you'll best any second-rate application, period. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] http: libevent vs many threads
On Fri, Apr 18, 2008 at 10:36:53PM +0200, Cezary Rzewuski wrote: Thank you for Your suggestions. I've just finished the implementation. I used the approach of libevent as HTTP server and threads working on downloaded content (they are performing some statistical computation on downloaded javascripts). It looks to work efficiently. It's probably not the right group, but you says that switching between threads is expensive. However, I've read somewhere (it was probably Advanced linux programming by Alex Samuel) that creating a new thread is nearly as fast as calling a function. Does it mean, that switching between threads is slower than creating a new thread? Creating a new thread is almost certainly not as fast as calling a function. Maybe, I suppose, as fast calling into the kernel; author's point being that Linux can quickly allocate and setup the task structures. But this sounds like one of those things you should test yourself. I'm not quite sure what the benchmark would look like. Try Usenet: comp.unix.programmer, or comp.programming.threads. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] performance patch
On Wed, Feb 27, 2008 at 02:54:41AM +0300, Vsevolod Stakhov wrote: Tani Hosokawa ?: snip significant amount of processing to do per request. The reason I put 1024 in there is all implementations that I know of silently truncate the backlog parameter to the system's maximum. The reason I don't use SOMAXCONN is some OS (older Solaris for sure) are incorrect for backward compatibility reasons, and other OS are incorrect because the actual setting is dynamically tunable via sysctl. snip As POSIX requires SOMAXCONN to be defined in sys/socket.h it is safe to use this value or define some constant in configure script, that detects SOMAXCONN definition. Also in old Solaris systems backlog was limited to 5. The problem with SOMAXCONN is the same as FOPEN_MAX (ISO C). The runtime limit is not the same as the compile-time constant. In fact, usually the runtime limit is much higher. On all of my boxes FOPEN_MAX is defined to be 16 or 20. Yet that is definitely not the runtime limit. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] SIGPIPE in multithreaded program
On Tue, Jan 22, 2008 at 08:51:55AM +0100, Ron Arts wrote: snip Oops, I'm sorry, I did not make myself clear, while writing the email I edited it a lot, and forgot to mention that indeed I ignore SIGPIPE in my initialisation code: snip But my program is still being killed with SIGPIPE occasionally. I am using threads, and I presume sometimes one of the other threads receives the SIGPIPE signal instead of the main thread, and I *think* that in such a case my program exits. But what I meant to ask was: isn't libevent supposed (since 1.3) to handle multithreading and ensure that only one thread receives the signal? Neither libevent nor anything else can specify that any particular thread receive a signal. All that can be done is to block a signal in all threads except for one, which gives the same effect in a round-about manner. Since libevent doesn't create the threads, it can't block them like this. And, in any event, it would be evil for libevent to fiddle with signals this way behind a process' back. I wouldn't want libevent to preemptively block all signals from event_init(). Or should I specifically add code at the beginning of each thread to ignore SIGPIPE? Signal handlers and masks are inherited across fork() and pthread_create(). And masks are inherited across exec(), too, I think. You could add code to each thread to do this. Of course, there are race conditions. Imagine if a signal is raised after a thread starts but before it can block the signal. This is especially troublesome if you dynamically create threads. With SIGPIPE the answer is simple, though. Block the signal from the main thread before creating any other threads. All threads will inherit the block, and SIGPIPE can never squeeze through. But, by block I mean actually using sigaction(2) (see the code I posted earlier), not by installing a libevent handler. Installing a SIGPIPE handler through libevent is pointless and a waste of CPU, and of course it doesn't do what you want anyway. Signals are probably the most complex and difficult to understand concept in Unix, and understanding how to safely use them is even more difficult. I suggest you find a copy of Richard Stevens' Advanced Programming in the UNIX Environment. If Knuth and Stevens have (had) one thing in common, it's that no engineer would ever question why you one of their books sat on your book shelf. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] SIGPIPE in multithreaded program
On Tue, Jan 22, 2008 at 02:08:40PM +0100, Hannah Schroeter wrote: snip With SIGPIPE the answer is simple, though. Block the signal from the main thread before creating any other threads. All threads will inherit the block, and SIGPIPE can never squeeze through. I think you mean *ignore* SIGPIPE. Right. It's late. (Or was late; now it's morning again.) At this point I've lost enough points that I ought to take a refresher course myself ;) ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] thread-safety (and performance)
On Mon, Jan 21, 2008 at 04:14:08PM -0800, Tani Hosokawa wrote:
snip
@@ -1999,7 +1999,7 @@
if ((fd = bind_socket(address, port)) == -1)
return (-1);
- if (listen(fd, 10) == -1) {
+ if (listen(fd, 8192) == -1) {
event_warn(%s: listen, __func__);
EVUTIL_CLOSESOCKET(fd);
return (-1);
Probably better to use SOMAXCONN instead of an arbitrary number.
___
Libevent-users mailing list
Libevent-users@monkey.org
http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] SSL enabling libevent programs, is there an example available?
On Tue, Jan 22, 2008 at 01:09:11AM +0100, Ron Arts wrote: Hi, subject says it all. If there's no example with libevent, can someone recommend another place to look? I tried Googling but couldn't find a concise example. http://25thandclement.com/~william/projects/libevnet.html Download the tarball. OpenSSL bindings are in src/tls.c. There's lots more code in the library for integrating with libevent, but I wrote tls.c many years ago (before libevnet, and possibly before libevent), and it should be easy to rip it out and use however you want. At the very least, tls_read_try(), tls_write_try() and tls_accept_try() should help you out. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] [PATCH] relative timer with EV_PERSIST support + move event_del logic into event_active + regression tests
On Tue, Nov 13, 2007 at 06:15:00PM -0800, Christopher Layne wrote: 1. Make EV_PERSIST reschedule timeouts automatically. 2. New function: timeout_schedule: (nothing really new within it, just modular wrapping of timeout rescheduling). 3. New macro: evutil_timercpy: tv_dst = tv_src in one line type deal; 3. Regression tests for persistent timeouts, include read/write, signals, and timers. 4. Another regression test for signal handler restores (no problem, just added another one). So what this means is that if you do the following: event_set(ev, fd, EV_READ | EV_PERSIST, read_cb, obj); event_add(ev, timeout); read_cb() will be called whenever a read event happens, and it's timeout as passed to event_add() will be reset to the original value you passed. You do not have to call event_add() within the handler. event_set(ev, -1, EV_TIMEOUT | EV_PERSIST, timer_cb, obj); event_add(ev, cycle); timer_cb() will be called when timeout (as passed via cycle) expires. It will then reschedule itself with it's original timeout, e.g. periodic timer. You do not have to call event_add() within the handler. What was the original behavior? Did the timer disappear? If not, I fail to see how the new behavior is more justificed than the old; meaning, instead of changing it, why not add it as an option? If waiting to read an entire line, but you are trickled one byte at a time, do you really want to reset the timeout between bytes, or is the timeout meant to limit the time spent on reading the line? For the event_del() changes, it's just moving event_del() into event_active(), when an event occurs. There is a feature request on sourceforge for this, and this couples nicely with the EV_PERSIST change. It also allows us to rexamine the logic tests within the various event dispatchers themselves, as event_active() will only delete the same event once. Will this cause incompatabilities? What if you call event_add() anyhow? Will it return failure? This could silently break code. Maybe the user set EV_PERSIST, realized it didn't do what he wanted, but never removed it. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkeymail.org/mailman/listinfo/libevent-users
Re: [Libevent-users] sensible thread-safe signal handling proposal
On Sun, Nov 04, 2007 at 03:18:42PM -0800, Steven Grimm wrote: You've just pretty accurately described my initial implementation of thread support in memcached. It worked, but it was both more CPU- intensive and had higher response latency (yes, I actually measured it) than the model I'm using now. The only practical downside of my current implementation is that when there is only one UDP packet waiting to be processed, some CPU time is wasted on the threads that don't end up winning the race to read it. But those threads were idle at that instant anyway (or they wouldn't have been in a position to wake up) so, according to my benchmarking, there doesn't turn out to be an impact on latency. And though I am wasting CPU cycles, my total CPU consumption still ends up being lower than passing messages around between threads. Is this on Linux? They addressed the stampeding herd problem years ago. If you dig deep down in the kernel you'll see their waitq implemention for non-blocking socket work (and lots of other stuff). Only one thread is ever woken per event. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] announcing libev, towards a faster and more featureful libevent
On Sat, Nov 03, 2007 at 09:15:07PM +0100, Marc Lehmann wrote: snip In that mail, I announced that I will work on the problems I encountered in libevent (many of which have been reported and discusssed earlier on this list). After analyzing libevent I decided that it wasn't fixable except by rewriting the core parts of it (the inability to have multiple watchers for the same file descriptor event turned out to be blocking for my applications, otherwise I wouldn't have started the effort in the first place...). A good itch, indeed. The results look promising so far: I additionally implemented a libevent compatibility layer and benchmarked both libraries using the benchmark program provided by libevent: http://libev.schmorp.de/bench.html Here is an incomplete list of what I changed and added (see the full list at http://cvs.schmorp.de/libev/README, or the cvs repository at http://cvs.schmorp.de/libev/): Man. More pressure to rename my library from libevnet to something else ;) snip * there is full support for fork, you can continue to use the event loop in the parent and child (or just one of them), even with quirky backends such as epoll. Curious how you managed to do this. Are you checking the process PID on each loop? * there are two types of timers, based on real time differences and wall clock time (cron-like). timers can also be repeating and be reset at almost no cost (for idle timeouts used by many network servers). time jumps get detected reliably in both directions with or without a monotonic clock. But then they're not truly real-time, no? snip * event watchers can be added and removed at any time (in libevent, removing events that are pending can lead to crashes). This is news to me. Can you give more detail, maybe with pointers to code? * different types of events use different watchers, so you don't have to use an i/o event watcher for timeouts, and you can reset timers seperately from other types of watchers. Also, watchers are much smaller (even the libevent emulation watcher only has about 2/3 of the size of a libevent watcher). libevnet does this for I/O; timer is always set separately from read/write events. (Point being, its using libevent.) * I added idle watchers, pid watchers and hook watchers into the event loop, as is required for integration of other event-based libraries, without having to force the use of some construct around event_loop. Needing to do an operation on every loop is arguably very rare, and there's not much burden in rolling your own. PID watchers, likewise... how many spots in the code independently manage processes (as opposed to one unit which can just catch SIGCHLD). Also, curious how/if you've considered Win32 environments. * the backends use a much simpler design. unlike in libevent, the code to handle events is not duplicated for each backend, backends deal only with file descriptor events and a single timeout value, everything else is handled by the core, which also optimises state changes (the epoll backend is 100 lines in libev, as opposed to 350 lines in libevent, without suffering from its limitations). libevnet optimizes state changes. Logically every I/O request is single-shot (which is more forgiving to user code), but it actually sets EV_PERSIST and delays libevent bookkeeping until the [libevnet bufio] callback returns. If the user code submits another I/O op from its callback (highly likely) then the event is left unchanged. It's still re-entrant safe because it can detect further activity up the call chain using some stack message passing bits (instead of reference counting because I also use mem pools, but I digress). Again, point being this can be done using libevent as-is. As for compatibility, the actual libev api is very different to the libevent API (although the design is similar), but there is a emulation layer with a corresponding event.h file that supports the event library (but no evbuffer, evnds, evhttp etc.). Well... if you can persuade me of the utility then this Christmas I might want to investigate writing an evdns-like component. See the lookup component of libevnet. There are lots of nice things I need in a DNS resolver that evdns and others are incapable of handling. And I've also written more HTTP, RTSP, and SOCKS5 parsers than I can remember. snip The obvious plan would be to take the evhttp etc. code from libevent and paste it in to libev, making libev a complete replacement for libevent with an optional new API. The catch is, I'd like to avoid this, because I am not prepared to maintain yet another library, and I am not keen on replicating the configure and portability work that went into libevent so far. If you ask me, it would prove more fortuitous to re-write the DNS and HTTP components then to replace libevent. Reason being because it would be hard to substantively improve on DNS/HTTP without altering the API, whereas
Re: [Libevent-users] file io
On Tue, Oct 09, 2007 at 11:06:21AM -0400, arthur zhang wrote: Hi All, As my understanding, with libevent, any io should be non_blocking, and here comes my dumb q;-) I want log stuffs(trace) to a disk file in my comm callback, so I open a file like below, but I didn't get any write event to call back. I am using Ubuntu Dapper and my target system is Solaris 10. 1) You can't do non-blocking file I/O in Unix using the standard interfaces. You need to use a horrible and poorly implemented interfaces like POSIX Asynchronous I/O, which is impossible to use with libevent. 2) A file descriptor (to a realfile) will _always_ poll as ready for reading or writing. (The logic being, even though the disk might be slow, it's still always available for I/O--the disk is one gigantic buffer.) Thus, if your code was written properly your complaint should have been that you got an endless succession of write readiness callbacks suspicously lacking any gap. Using stdout worked. If stdout was a pipe, then it would work. Thus, you could shamefully (but with good reason) pipe to cat, and redirect cat's stdout to a file. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] EV_PERSIST behavior
On Tue, May 08, 2007 at 06:10:29PM -0700, Christopher Layne wrote: On Tue, May 08, 2007 at 02:03:17PM -0700, Niels Provos wrote: On 5/8/07, Phil Oleson [EMAIL PROTECTED] wrote: So.. To fix your implementation you will need to do something like I did. I reimplemented libevents' gettime() function (because it's not exposed via event.h) and use it instead of calling time(NULL); I don't really understand why you are saying that. Timeouts are incremental and not in absolute time. So, you don't really need to understand anything about the underlying implementation. To me, it seems like the timeouts do not reset on an EV_PERSIST event that received a non-timeout event. In a way this is absolute, not in the epoch sense, but that it's tied to the time event_add() was called and not relative to when the last valid event was received. That may or may not make sense the majority of the time. Most of the time you're reading logical data atoms, not bytes. Say you set a timeout for 10 seconds, but a socket polled as ready once every 9 seconds. Maybe there's a single byte available, maybe there's none (because the event notification was spurious). You're line buffering, and maybe you've set an upper bound on line length of 1000 bytes (e.g. SMTP). You could be polling for almost 3 hours if the peer trickled data byte-by-byte. If the peer was exceptionally smart, he could keep the connection open forever, by sending malformed packets which set the socket state as ready but were subsequently discarded further down the TCP stack. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] Re: IOCP writes
On Thu, Feb 22, 2007 at 12:09:18AM +0100, Rhythmic Fistman wrote: Issuing simultaneous asynchronous reads and writes on socket is a tiny bit rare and special. Are you sure that's what you want to do? If so, great - io completion queues can handle it. Bu-ut if libevent was modelled on the unix select-style interfaces, most of which don't easily support this kind of thing, how is this situation cropping up, assuming that this is just an IOCP port of libevent? Almost all of my use of libevent involves simultaneous reads/writes, for relays which filter streams on-the-fly, in both directions. It was tricky at first, but over the course of several revisions it all settled out. *sigh* Someday I'll have to muster up the energy to change the name, because on this list its just damn confusing, but my buffered-IO implementation is in libevnet: http://www.25thandclement.com/~william/projects/libevnet.html Specifically, see src/bufio/socket.c and the functions bufio_socket_event_run bufio_socket_event_mod bufio_socket_event_del bufio_socket_event_add bufio_socket_poll_handler bufio_sink_poll_handler Cancellations were tricky. Also, what I fail to see in most async libraries is logic to handle re-entrancy from recursive callbacks. That's what the FRAME_PUSH/FRAME_POP calls are all about. evdns.c fails in this respect, last time I looked, as does c-ares; they'll just segfault. My lookup API, built around c-ares, works around the c-ares problem w/ what I consider an abysmal hack; I'm looking to maybe fork the guts into libevnet because my re-entrancy patches were rejected. Anyhow, lookup.c does much, much more than c-ares, ADNS, or evdns, combined. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] [Patch] Install autoconf header file as evconfig.h
On Fri, Feb 16, 2007 at 07:17:17PM -0500, Nick Mathewson wrote: snip I think your approach is reasonable too in the medium term; you should probably send a patch to the list too, so yours can win. :) *yawn* I'd rather complain from the sidelines whilst other people do the work ;) If I can keep a strong stomach tonight I might dive into the autoconf morass. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] [Patch] Install autoconf header file as evconfig.h
On Fri, Feb 16, 2007 at 04:58:53PM -0800, Dave Gotwisner wrote: Yes, C doesn't define it (ANSI should have at the char/short/long/long long level). Even though there is nothing requiring that short is 16 and long is 32, the fact that the type.h files I mentioned in my previous post imply that they really are required to be these sizes, or the header files be broke. But C *does* define fixed-width types, through stdint.h, ever since C99 (going on 8 years, now!). And all modern platforms which even contemplate some POSIX/SUSv3 support, provide stdint.h, now, w/ the glaring exception of Microsoft and the Visual Studio suite. Actually, I think that SUSv3 mandates uint8_t, uint16_t, uint32_t, similar to how POSIX/SUS mandates that CHAR_BIT is 8. C99, strictly speaking, only requires the fixed-width types be defined through stdint.h if they can be represented as such. C99 also defines least and fast types. In any event, it kills me that Microsoft won't even add stdint.h. Kills me! People rail against C99 all the time, but the fact is that some very important parts are very nearly universally supported, now, at least in terms of number of distinct enviornments. stdint.h and snprintf() are the two most widely known. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] struct addrinfo is missing in cygwin's /usr/include/netdb.h
On Fri, Jan 26, 2007 at 01:49:50PM +0800, Hor Meng Yoong wrote: Hi: I tried to compile libevent-1.2a on cygwin. The structure addrinfo is not defined in /usr/include/netdb.h. Anyone has a fix for this? Cygwin hasn't added it yet. You might try grabbing the portable version from the OpenSSH sources. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] implementing timers in a multithread application
On Mon, Jan 22, 2007 at 02:06:10AM +0200, Rafi Cohen wrote: Hello, I'm new in this list and I apologize in advance if my questions are too basic. I'm writing an application which talks with a number of test devices connected to a gpib card and I'm using for this the open source linux gpib driver (http://linux-gpib.sourceforge.net). This is a multithread application, each thread for each device as each device has it's own device descriptor to read from and write to it. Within various states of the state machine of each thread, I need to implement a look, sending a query to the device for it's current state and reading the response. The loop may end in 2 ways: either the device has transitioned to a new state or timeout has occured and no transition happened during this time. Those timers are my problem here. The timers may vary within a thread, depending on the expected state and, each thread has to have it's own timers independent of the other threads. I cannot implement this by using alarm, setitimer and getitimer as they are process wide and not thread safe. So I wonder if libevent may be the appropriate tool to help me implement those timers. It sounds like libevent is not what you need. For libevent to even be an option in your scenario, your device descriptors must be pollable and capable of operating in non-blocking mode. But, if they're pollable that begs the question of why you're using an asynchronous alarm. Assuming you require the asynchronous alarm/timer, why not do it yourself? You only need one timer for the entire process. For example, install a timer with an interval which is the minimum precision you would need to service a timeout for any blocking operation in any of your threads. Block that interval timer's signal in all threads but the alarm multiplexing thread. Then, manually call pthread_kill()--from the multiplexing thread--for each thread which requested a timeout. Using, of course, a different, non-blocked signal number so the other threads can receive it. Of course, the implementation isn't trivial. Dealing with signals alone, or threads alone, is difficult enough. Running a libevent loop in the multiplexing thread in order to take advantage of the signal synchronizing behavior (libevent sends signals down a pipe(2) so they can be dealt with more safely) might be an option. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] Newbie questions
On Tue, Oct 24, 2006 at 07:48:10AM +0800, Adrian Chadd wrote: On Mon, Oct 23, 2006, William Ahern wrote: Note that with poll and select you don't lose much by shifting an event in and out of a polling state. However, with things like epoll() and kqueue() you potentially incur a system call everytime you do switch the notification state of each descriptor, which is costly. From my reading of the kqueue code this isn't the case. The change is 'bundled up' into the 'changes' array and kept there until the next call to kq_dispatch(). Yet another reason why kqueue is superior to epoll? ;) ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] libevent 1.2
On Wed, Oct 18, 2006 at 12:04:11AM -0600, Jason Ish wrote: I ran across at least one criticism of libevent gaining too much other than pure event library: http://varnish.projects.linpro.no/wiki/EventLibrary All of these criticisms are partially or wholly invalid. The DNS and HTTP code is effectively only distributed in the tarball; it's not part of the library itself. I don't understand the issue with struct event_base. Obviously it evolved, but I don't understand how the current API is any less useful to a threaded application. The timestamp issue sounds like, it doesn't have my particular pet feature, so the library is misdesigned. First of all, why is anybody depending on timestamps in such a pervasive manner as to want it built into the libevent API, knowing or not knowing that the timestamp may or may not be the particular timestamp the application desires (timestamp on a signal delivered through a pipe flushed at the end of a loop... how is that remotely useful for something seriously depending on a timestamp?). I *have* written code to give back a fixed timestamp per loop, and maybe it would be convenient if I could access libevent's copy, but I wouldn't criticize libevent for that. And where are these bugs and misfeatures in libevent? Personally I don't like the model used for the evbuffer framework, but I wouldn't call it a bug or misfeature. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] Can't get libevent ( event-test.c ) to work on OS X ( 10.4.8 )!
On Mon, Oct 09, 2006 at 11:41:52AM -0700, Niels Provos wrote: You should really just use the code in regress.c event-test.c is very old. BTW, the open O_NONBLOCK behavior is different across the different operating systems. This code works fine on *BSD systems. huh. interesting. learn something new every day. i should've stuck to testing my conclusions in code before replying. also, i didn't realize event-test.c was *the* event-test.c from the library. that wouldn't changed my operating assumptions a little ;) ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] Multithreaded behavior
On Fri, Sep 08, 2006 at 10:18:11AM -0700, Scott Lamb wrote: At a high level, I think it would require the basic poll algorithm to be: lock loop: while there are events: dequeue one unlock handle it lock if someThreadPolling: condition wait else: someThreadPolling = true poll for events lock fire condition unlock so whatever thread happens to notice that it's out of events does a poll, and the others can see the results immediately. But I haven't addressed actually putting new fds into the poll array. I'm not sure what the behavior there has to be. I admit it - this approach is complicated. Ah. I was approaching it from another angle (one thread per event loop, and the question being how to inject events and balance events into each event loop). I'd never want to touch the above scheme w/ a ten foot pole, mostly because one of the greatest benefits I enjoy with event-oriented programming is the lack of contention (i.e. not having to use a mutex everywhere). There are lots of scenarios where I might have multiple events outstanding, all related to a single context (TCP proxying, for instance). In the above design, I'd have to begin littering mutexes around my code. Relating those oustanding events to a shared event loop implicitly frees me from having to deal w/ this problem. When I use threads the only mutexes I want are the ones protecting malloc()/free(), and the handful of other similar global resources. Anyway, I'm not suggesting adopting this without actual proof that it's better. I need to blow the dust off my benchmark tools, but I'm willing to put the effort into trying things out if I hear ideas I like. Fair enough. ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] Multithreaded behavior
On Fri, Sep 08, 2006 at 04:17:20PM -0700, Scott Lamb wrote: On Sep 8, 2006, at 12:13 PM, William Ahern wrote: Ah. I was approaching it from another angle (one thread per event loop, and the question being how to inject events and balance events into each event loop). Like the first thing I described? Have you actually done this and had any luck with it? I suppose I could give it a go, at least for a simple balancing scheme. What I've done in an MTA is use descriptor passing. A master process listens for connections and than sends them to it's children. I keep a tree of children, ordered by number of outstanding connections. When a child loses a connection it sends a message down a pipe so I can decrement it's connection count. But that's sort of heavy weight; ultimately I liked the idea because it provided robustness. I would like to try this scheme in a single-process, multiple thread environment. Each event loop could have an outstanding event listening on a pipe (just like the signal pipe in libevent). The thread accepting connections would select a thread to hand-off a connection and dump it into a queue, then signal that thread's event pipe. Actually, this is how I had assumed everybody else was doing it after I saw the event_base support go into the library. And also the source of my poll on a mutex, because using a pipe for this also always seemed a little heavy weight. Though, I've never benchmarked so I have no right ;) ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
Re: [Libevent-users] strange bufferevent_new interface
On Thu, Jul 27, 2006 at 09:27:09PM +0200, Rainer Giedat wrote: Hi, while reading some code which used buffered events, i saw that you really have to define callbacks for read, write and error, even if you do not care about one of them. Is there a reason for that? This ends up in ugly code, defining empty functions as callbacks. The problem is, that bufferevent_new enables EV_WRITE and EV_READ regardless of beeing defined not NULL. (On OpenBSD only EV_WRITE currently) If there is no reason for that, i would suggest to apply the following diff. It fixes the problem for me. :) Thanks for the great work Niels. So long... Well, w/o this behavior it might be impossible to integrate this w/ OpenSSL. Just to complete a read through an SSL/TLS stream, you may need to first complete a write (for instance, when re-keying or other protocol management work) which can actually occur quite often. I never figured out how to make this work cleanly w/ libevent's buffered I/O API. Maybe it's just not possible, period, so it doesn't matter. - Bill ___ Libevent-users mailing list Libevent-users@monkey.org http://monkey.org/mailman/listinfo/libevent-users
