Re: [PATCH] calculate fudge on win32
On Fri, Jun 3, 2011 at 3:01 AM, Marc Lehmann wrote: > Can you try this patch on your 64hz windows? If that works, that (or > soemthing close using e.g. Sleep) will go into the next version of libev. > > http://ue.tst.eu/1f5a35969eb478f97d5e2ba8c7537266.txt Yes, thank you. It works as expected. On the other hand since libev provides ev_sleep as part of its API I think the check of delay being too small (or using ceil instead of relying on default runtime settings) should go there. :) ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
[PATCH] calculate fudge on win32
On Fri, Jun 3, 2011 at 12:34 AM, Marc Lehmann wrote: > As for 0, we indeed don't need to care, but introducing this special case > would make the timeout values nonproportional to the actual time. And that > in turn would destroy the property that adding something to a timeout > value will make the timer fire this much earlier or later (within the > limits of libev's control). > > I think having time run strictly monotonously is more important than the > special case of a 0-second timer - a 0 second timer is definitely NOT a > special case for the user of libev. > > Right. I agree this property shouldn't become broken. > > It is possible to use ev_timer(-1) but that's not as clear as > ev_timer(0). > > Why would anybody want to use specifically a 0-second timer (where the 0 > isn't just the chance result of some calculation)? > > The only reason I can know from past experience is that some people try > to abuse 0-second timers as some replacement for idle watchers, possibly > because that hack was the only way to get some kind of idle behaviour with > some other event loop. > Well, in gevent, people sometimes use sleep(0) as a way to yield to other current coroutines. However, I can also handle 0 separately in sleep(), make it use ev_idle or ev_check in that case. ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: updating timers safely
thank you very much it is more clear now. On 2 June 2011 21:28, Brandon Black wrote: > On Thu, Jun 2, 2011 at 10:25 PM, Brandon Black wrote: > > On Thu, Jun 2, 2011 at 10:15 PM, Juan Pablo L > > wrote: > >> i m trying to avoid using start/stop or even ev_timer_again because the > >> socket is considered busy when taken by a thread and thread will hold > the > >> socket for as long as clients have something to send which could be a > lot of > >> info > >> or just a few bytes, i m trying to say that a thread may hold the socket > >> longer than the > >> idle time out so how can i manage > >> to prevent the event loop from disconnecting a socket that is being used > >> by another thread ? > > > > Well, not knowing the rest of this design, my guess would be that when > > a thread takes control of a socket, it should remove it from the idle > > timer linked list, and when it's done it should re-add it at the end. > > > > I should add: you might try a single-thread design first. Seems like > every time I design for multi-threaded workers to take advantage of > CPU cores, I end up hitting some other limit before I need them and > going back to just running one thread. My last project, I maxed out > the packets/sec rate on the network interface before I had to spin up > a second worker thread. But then again, most of these projects I'm > talking about had very light real processing requirements, they were > mostly just shuffling data around (e.g. proxy servers for custom > protocols). > ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: updating timers safely
On Thu, Jun 2, 2011 at 10:25 PM, Brandon Black wrote: > On Thu, Jun 2, 2011 at 10:15 PM, Juan Pablo L > wrote: >> i m trying to avoid using start/stop or even ev_timer_again because the >> socket is considered busy when taken by a thread and thread will hold the >> socket for as long as clients have something to send which could be a lot of >> info >> or just a few bytes, i m trying to say that a thread may hold the socket >> longer than the >> idle time out so how can i manage >> to prevent the event loop from disconnecting a socket that is being used >> by another thread ? > > Well, not knowing the rest of this design, my guess would be that when > a thread takes control of a socket, it should remove it from the idle > timer linked list, and when it's done it should re-add it at the end. > I should add: you might try a single-thread design first. Seems like every time I design for multi-threaded workers to take advantage of CPU cores, I end up hitting some other limit before I need them and going back to just running one thread. My last project, I maxed out the packets/sec rate on the network interface before I had to spin up a second worker thread. But then again, most of these projects I'm talking about had very light real processing requirements, they were mostly just shuffling data around (e.g. proxy servers for custom protocols). ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: updating timers safely
On Thu, Jun 2, 2011 at 10:15 PM, Juan Pablo L wrote: > i m trying to avoid using start/stop or even ev_timer_again because the > socket is considered busy when taken by a thread and thread will hold the > socket for as long as clients have something to send which could be a lot of > info > or just a few bytes, i m trying to say that a thread may hold the socket > longer than the > idle time out so how can i manage > to prevent the event loop from disconnecting a socket that is being used > by another thread ? Well, not knowing the rest of this design, my guess would be that when a thread takes control of a socket, it should remove it from the idle timer linked list, and when it's done it should re-add it at the end. ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: updating timers safely
Thanks a lot, that is a bit clearer, so the event loop will always be controlling the first timer only, which is supposed to be the closest to expiration (and some few next nodes) right ? i m trying to avoid using start/stop or even ev_timer_again because the socket is considered busy when taken by a thread and thread will hold the socket for as long as clients have something to send which could be a lot of info or just a few bytes, i m trying to say that a thread may hold the socket longer than the idle time out so how can i manage to prevent the event loop from disconnecting a socket that is being used by another thread ? what i mean is, if every time the socket is taken by a thread i will update the timer to expire later (now() + timeout) and put it at the end of the list, there is no way for me to know how long the socket will be taken by the thread and thus busy (not idle) ? (i would not like to use semaphores or mutexes, etc) My background with timers is limited only to a few timers so i did not mind to start/stop in the past, but this time that does not apply. On 2 June 2011 20:59, Brandon Black wrote: > On Thu, Jun 2, 2011 at 9:08 PM, Juan Pablo L > wrote: > > thanks for your reply, actually yes i have read the part about the Be > smart > > about timer, > > but this is the first time i have to manage so many timers, and something > is > > not clear to me. > > > > #4 refers to making a linked list and putting all timers in there but how > > can i control individually > > each timer, [] > > It works, as long as you maintain the list in timer order, which is > O(1) if all the timeout intervals are the same. When a connection > enters its "connected" state where you want to start the idle timer, > you simply add it to the tail of the list. When a connection ends, > you remove it from the list. When a connection has an I/O event to > reset the idle timer, you move it to the bottom of the list. > > The actual ev_timer is set for the time the head of the list needs to > fire. When it fires, you handle the connection at the head (and any > more in list order within some small epsilon past the current > ev_now()). If "handling" the timeout at the top of the list means > killing the connection, then it simply gets removed from the list. If > not, it goes to the bottom (e.g. a recurring maintenance timer > per-connection, as opposed to an idle-kill timer). At the end of the > callback, you set the next timeout for the timer to the time of the > new list head. > > The details can be tricky to implement and debug. IMHO as the docs > state, it's only worth it if you have a *ton* of identical timeouts. > It would be neat if someone wrote a generic implementation in C that > worked with arbitrary user-defined connection datatypes while keeping > list management pointers inside them. > > -- Brandon > ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: updating timers safely
On Thu, Jun 2, 2011 at 9:08 PM, Juan Pablo L wrote: > thanks for your reply, actually yes i have read the part about the Be smart > about timer, > but this is the first time i have to manage so many timers, and something is > not clear to me. > > #4 refers to making a linked list and putting all timers in there but how > can i control individually > each timer, [] It works, as long as you maintain the list in timer order, which is O(1) if all the timeout intervals are the same. When a connection enters its "connected" state where you want to start the idle timer, you simply add it to the tail of the list. When a connection ends, you remove it from the list. When a connection has an I/O event to reset the idle timer, you move it to the bottom of the list. The actual ev_timer is set for the time the head of the list needs to fire. When it fires, you handle the connection at the head (and any more in list order within some small epsilon past the current ev_now()). If "handling" the timeout at the top of the list means killing the connection, then it simply gets removed from the list. If not, it goes to the bottom (e.g. a recurring maintenance timer per-connection, as opposed to an idle-kill timer). At the end of the callback, you set the next timeout for the timer to the time of the new list head. The details can be tricky to implement and debug. IMHO as the docs state, it's only worth it if you have a *ton* of identical timeouts. It would be neat if someone wrote a generic implementation in C that worked with arbitrary user-defined connection datatypes while keeping list management pointers inside them. -- Brandon ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: updating timers safely
thanks for your reply, actually yes i have read the part about the Be smart about timer, but this is the first time i have to manage so many timers, and something is not clear to me. #4 refers to making a linked list and putting all timers in there but how can i control individually each timer, even though they all will have the same time out not all of them will be used the same quantity of times, each timer will have a socket associated to it, so some timers will be updated more times than others and others will expire more times than others depending on the activity in the sockets, others may not even time out at all, depending on the usage of the socket. are you aware of an example somewhere that uses timer using technique #4 ? i m sure i m missing something. thanks!!! On 2 June 2011 17:37, Marc Lehmann wrote: > On Thu, Jun 02, 2011 at 04:59:22PM -0600, Juan Pablo L < > jpablolorenze...@gmail.com> wrote: > > Any tip would be appreciated on how to deal with this situation, maybe > > You claim to have read the documentation but managed to miss the "Be smart > about timeouts" section? > > > i may have thousands if not millions of sockets and timers (one timer per > > socket) > > If your timeouts are all constant, then method #4 would be O(1) and very > fast, too. > > If any questions are left after reading the docs, feel free to ask them :) > > -- >The choice of a Deliantra, the free code+content MORPG > -==- _GNU_ http://www.deliantra.net > ==-- _ generation > ---==---(_)__ __ __ Marc Lehmann > --==---/ / _ \/ // /\ \/ / schm...@schmorp.de > -=/_/_//_/\_,_/ /_/\_\ > ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: updating timers safely
On Thu, Jun 02, 2011 at 04:59:22PM -0600, Juan Pablo L wrote: > Any tip would be appreciated on how to deal with this situation, maybe You claim to have read the documentation but managed to miss the "Be smart about timeouts" section? > i may have thousands if not millions of sockets and timers (one timer per > socket) If your timeouts are all constant, then method #4 would be O(1) and very fast, too. If any questions are left after reading the docs, feel free to ask them :) -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schm...@schmorp.de -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: [PATCH] calculate fudge on win32
On Thu, Jun 02, 2011 at 10:25:56AM +0400, Alexey Borzenkov wrote: > Which is also why calculating fudge the way I did is meaningless, > because if it happens to be smaller than 1e-3 it would fold into > Sleep(0), and remember that by my calculations on Windows 7 ev_time > resolution is 5e-4, so it would cause problems. > > Here's the obvious patch: Can you try this patch on your 64hz windows? If that works, that (or soemthing close using e.g. Sleep) will go into the next version of libev. http://ue.tst.eu/1f5a35969eb478f97d5e2ba8c7537266.txt -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schm...@schmorp.de -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
updating timers safely
Hi, i need to disconnect a socket after some time of inactivity, i m not sure how to do that in an optimum way( according to the docs stopping/starting every time is not optimum) and safely (so updating the repeat member from a different thread while the socket is being under the control of the loop thread). and also i do not want the loop to be disconnecting the socket while some threat is using the socket, so i dont know how much i would increase the repeat member if it was safe to do it from another thread any ways. Any tip would be appreciated on how to deal with this situation, maybe you think the only thing i can do is just to start/stop the sockets every time ? i may have thousands if not millions of sockets and timers (one timer per socket) and they might be used by a thread for an unknown period of time before being available again for another thread to use it. thanks!! ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: [PATCH] calculate fudge on win32
On Thu, Jun 02, 2011 at 09:51:13PM +0200, Denis Bilenko wrote: > > + backend_fudge = 1e-3; > > > > This helps with CPU load but it still causes ev_timer(0) to issue > ev_sleep(>0) which puts the process to sleep. As it should, see below. > However, for ev_timer with AT = 0, the guarantee that the timers are only > executed after AT seconds is passed is satisfied immediately, No, "right now" is not also "past now". The reason why "past" is so important, imagine your clock had only a 1 seocnd resolution. Imagine at one point it is second 1000, and then you start a 1 second timeout. When the clock then turns to 1001 any amount of time from 0 to 1 second might have been passed, breaking the "at least" guarantee of libev. In other words, the only way to implement "at least" behaviour is to fire "past this time" not "past or at this time", because it is, in general, impossible to have a notion of "this time" that isn't already outdated. As for 0, we indeed don't need to care, but introducing this special case would make the timeout values nonproportional to the actual time. And that in turn would destroy the property that adding something to a timeout value will make the timer fire this much earlier or later (within the limits of libev's control). I think having time run strictly monotonously is more important than the special case of a 0-second timer - a 0 second timer is definitely NOT a special case for the user of libev. > It is possible to use ev_timer(-1) but that's not as clear as ev_timer(0). Why would anybody want to use specifically a 0-second timer (where the 0 isn't just the chance result of some calculation)? The only reason I can know from past experience is that some people try to abuse 0-second timers as some replacement for idle watchers, possibly because that hack was the only way to get some kind of idle behaviour with some other event loop. The fix for that is an idle watcher, it has defined semantics for this case. -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schm...@schmorp.de -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: [PATCH] calculate fudge on win32
On Thu, Jun 02, 2011 at 10:25:56AM +0400, Alexey Borzenkov wrote: > fudge = 1e-6: > with ev_io: prepare called 128 times > without ev_io: prepare called 1765445 times Right, I forgot about sleep - I guess that can be specialcased though, so 1e-6 it is then. Thanks for trying it out and reporting your findings! -- The choice of a Deliantra, the free code+content MORPG -==- _GNU_ http://www.deliantra.net ==-- _ generation ---==---(_)__ __ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schm...@schmorp.de -=/_/_//_/\_,_/ /_/\_\ ___ libev mailing list libev@lists.schmorp.de http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
Re: [PATCH] calculate fudge on win32
On Thu, Jun 2, 2011 at 8:25 AM, Alexey Borzenkov wrote:
> + backend_fudge = 1e-3;
>
This helps with CPU load but it still causes ev_timer(0) to issue
ev_sleep(>0) which puts the process to sleep.
However, for ev_timer with AT = 0, the guarantee that the timers are only
executed after AT seconds is passed is satisfied immediately, and thus there
is no need to sleep.
It is possible to use ev_timer(-1) but that's not as clear as ev_timer(0).
What do you think about making a special case for ev_timer(at=0)?
Here's a possible patch:
diff -r 4e6eab41f320 libev/ev.c
--- a/libev/ev.cThu Jun 02 21:03:56 2011 +0200
+++ b/libev/ev.cThu Jun 02 21:47:49 2011 +0200
@@ -2241,9 +2241,12 @@
/* first reschedule or stop timer */
if (w->repeat)
{
- ev_at (w) += w->repeat;
+ if (ev_at(w))
+ev_at (w) += w->repeat;
+ else
+ev_at (w) = mn_now + w->repeat;
if (ev_at (w) < mn_now)
-ev_at (w) = mn_now;
+ ev_at (w) = mn_now;
assert (("libev: negative ev_timer repeat value found while
processing timers", w->repeat > 0.));
@@ -2761,7 +2764,8 @@
if (expect_false (ev_is_active (w)))
return;
- ev_at (w) += mn_now;
+ if (ev_at(w))
+ ev_at (w) += mn_now;
assert (("libev: ev_timer_start called with negative timer repeat value",
w->repeat >= 0.));
@@ -2802,7 +2806,8 @@
}
}
- ev_at (w) -= mn_now;
+ if (ev_at(w))
+ ev_at (w) -= mn_now;
ev_stop (EV_A_ (W)w);
@@ -2837,7 +2842,9 @@
ev_tstamp
ev_timer_remaining (EV_P_ ev_timer *w)
{
- return ev_at (w) - (ev_is_active (w) ? mn_now : 0.);
+ if (ev_at(w))
+ return ev_at (w) - (ev_is_active (w) ? mn_now : 0.);
+ return 0.;
}
#if EV_PERIODIC_ENABLE
___
libev mailing list
libev@lists.schmorp.de
http://lists.schmorp.de/cgi-bin/mailman/listinfo/libev
