Re: Scoreboard 3.0
On Fri, May 13, 2016 at 8:34 AM, Eric Covener wrote: > On Wed, May 11, 2016 at 6:39 PM, William A Rowe Jr > wrote: > > In the next iteration for trunk I see a value in having both the basic > > activity flag per worker thread so we can see those states at a glance > as we > > can today, as well as a summary table per process of the number of > > recognized workers and inactive Conn/req per process by state. > > I believe we have (most of?) this data in 2.4 server-status for event. > Yup, that's certainly a great start to what we want. My open question is still, whether administrators would want to see both the per-connection and the per-thread detailed score records on the same server. Once we are in agreement on the backports of scoreboard.c to 2.4.x branch, it would be a good time to introduce a new state between reading and writing, perhaps called loading or processing, which reflects the time between the start of the request handler, and the beginning of writing to the client. This would reflect both processing time, and glomming of any input body.
Re: Scoreboard 3.0
On Wed, May 11, 2016 at 6:39 PM, William A Rowe Jr wrote: > In the next iteration for trunk I see a value in having both the basic > activity flag per worker thread so we can see those states at a glance as we > can today, as well as a summary table per process of the number of > recognized workers and inactive Conn/req per process by state. I believe we have (most of?) this data in 2.4 server-status for event.
Re: Scoreboard 3.0
On May 11, 2016 4:16 PM, "Eric Covener" wrote: > > On Wed, May 11, 2016 at 8:28 AM, Stefan Eissing > wrote: > > I got some such feedback when h2 connections truly went async and no longer showed up as occupying a slot. I share your idea that in an event based server, threads are not the primary dimension but connections are. > > Threads (slots) are the primary dimension for what you need to monitor > to not get overrun. The state of connections NOT running on a thread > (keepalive, write completion, lingering close) seems to me to still be > a distant second for routine work. I see value in listing individual > write-completion requests, since they could be an issue from the users > POV but would be de-emphasized in current server-status. In the next iteration for trunk I see a value in having both the basic activity flag per worker thread so we can see those states at a glance as we can today, as well as a summary table per process of the number of recognized workers and inactive Conn/req per process by state. Those should be detailed whether or not an extendedstatus is enabled, or not. That table should help us understand the nature of any server overload.
Re: Scoreboard 3.0
On Wed, May 11, 2016 at 8:28 AM, Stefan Eissing wrote: > I got some such feedback when h2 connections truly went async and no longer > showed up as occupying a slot. I share your idea that in an event based > server, threads are not the primary dimension but connections are. Threads (slots) are the primary dimension for what you need to monitor to not get overrun. The state of connections NOT running on a thread (keepalive, write completion, lingering close) seems to me to still be a distant second for routine work. I see value in listing individual write-completion requests, since they could be an issue from the users POV but would be de-emphasized in current server-status.
Re: Scoreboard 3.0
On Wed, May 11, 2016 at 10:28 AM, Stefan Eissing < stefan.eiss...@greenbytes.de> wrote: > > > Am 11.05.2016 um 17:03 schrieb William A Rowe Jr : > > > > It seems like a lot of the scoreboard confusion lies in the event > > MPM lies in the work-performed-by-thread model of the thread > > based scoreboard. > > > > Let's call this existing model 'ExtendedStatus threads' > > > > What it seems some users and committers are looking for is > > a different model, let's call this 'ExtendedStatus connections'. > > I got some such feedback when h2 connections truly went async and no > longer showed up as occupying a slot. I share your idea that in an event > based server, threads are not the primary dimension but connections are. > I think they are both representations, I'm curious if there is a use case for configuring both in verbose/extended mode at the same time. > The information which thread handled a connection at which point in time > seems irrelevant. If a connection occupies a thread for more than x > seconds, it certainly becomes interesting. Also the waiting time, should > all threads be occupied, is interesting as well as statistics about how > many keepalive connections were closed early due to worker exhaustion. +1 - e.g. track the time at which the connection awoke based on the actual triggers in the pollset, even while the dispatcher is still trying to find a slot for each of the now-ready sockets. But until we poll again and we won't until we dispatch all of the sockets which just woke us, we won't be seeing other connections which are potentially now awake. > If we adopted an ExtendedStatus connections, the scoreboard > > table would be much larger than the number of event threads, > > and would report all of the open connections to the server (at > > least, all those that had passed the httpready threshold of our > > awareness). It can't be a simple ring, because the slots would > > expire at different times relative to their injection, so we would > > probably want to track the oldest request. The most optimal > > implementation would have score pools per-process so that > > connection score slot reuse is a simple thread lock, but we > > could alternately perform this tracking across processes. > > > > The first question that comes to mind is whether there is any > > value in supporting ExtendedStatus threads+connections? > > Two entire detailed lists residing in two different score shms? > > Note that the traditional thread status flag scoreboard would > > certainly be preserved, for a thread summary, even when the > > user chooses ExtendedStatus connections. > > > > Feedback and more ideas? > > It seems the information which request had been processed is so > interesting (which I totally destroyed in my ignorance) that scoreboard > should make every attempt to keep it until it is replaced by something new. > This is fine for the "static" HTML view that there is now. > > What I would find really cool is a feature that records connection/worker > changes and lets me play that back in a browser. Maybe even a live feed of > (a subset of) activities. Something that you see in the background on Star > Trek, only it would be real. > Turning the conventional per-thread score and per-connection score into two score-notice handlers, and allowing the user to then load other modules which register against the score notice hook, would accomplish this... It could record individual deltas or snapshots at timed intervals. Not sure > what is more feasible. I once did a status monitor of a system with various > props that reported green/yellow/red and had that in a horizontal scroll in > a browser. Customer was not happy as it became apparent how often his > system was in red though...but that cannot happen to us! > I'm thinking ActiveMQ or a similar message broker might be our friend here for creating such an app. Not "logging", per say, but every actual server connection/request state change notifications. Digging the idea :)
Re: Scoreboard 3.0
> Am 11.05.2016 um 17:03 schrieb William A Rowe Jr : > > It seems like a lot of the scoreboard confusion lies in the event > MPM lies in the work-performed-by-thread model of the thread > based scoreboard. > > Let's call this existing model 'ExtendedStatus threads' > > What it seems some users and committers are looking for is > a different model, let's call this 'ExtendedStatus connections'. I got some such feedback when h2 connections truly went async and no longer showed up as occupying a slot. I share your idea that in an event based server, threads are not the primary dimension but connections are. The information which thread handled a connection at which point in time seems irrelevant. If a connection occupies a thread for more than x seconds, it certainly becomes interesting. Also the waiting time, should all threads be occupied, is interesting as well as statistics about how many keepalive connections were closed early due to worker exhaustion. > If we adopted an ExtendedStatus connections, the scoreboard > table would be much larger than the number of event threads, > and would report all of the open connections to the server (at > least, all those that had passed the httpready threshold of our > awareness). It can't be a simple ring, because the slots would > expire at different times relative to their injection, so we would > probably want to track the oldest request. The most optimal > implementation would have score pools per-process so that > connection score slot reuse is a simple thread lock, but we > could alternately perform this tracking across processes. > > The first question that comes to mind is whether there is any > value in supporting ExtendedStatus threads+connections? > Two entire detailed lists residing in two different score shms? > Note that the traditional thread status flag scoreboard would > certainly be preserved, for a thread summary, even when the > user chooses ExtendedStatus connections. > > Feedback and more ideas? It seems the information which request had been processed is so interesting (which I totally destroyed in my ignorance) that scoreboard should make every attempt to keep it until it is replaced by something new. This is fine for the "static" HTML view that there is now. What I would find really cool is a feature that records connection/worker changes and lets me play that back in a browser. Maybe even a live feed of (a subset of) activities. Something that you see in the background on Star Trek, only it would be real. It could record individual deltas or snapshots at timed intervals. Not sure what is more feasible. I once did a status monitor of a system with various props that reported green/yellow/red and had that in a horizontal scroll in a browser. Customer was not happy as it became apparent how often his system was in red though...but that cannot happen to us! -Stefan
