Re: Measuring the page generation of the forum
On Saturday, 14 April 2012 at 03:31:29 UTC, Jesse Phillips wrote: I use the split-thread view, I haven't had long page loads so maybe I've miss understood what he means by a hiccup. But I have seen it take time to retrieve a message and display it to me. I would expect a page could be considered "loading" if it is still waiting on fetching the post from the NG. All posts are stored in a local SQLite database. Posts are retrieved from the NG when the program starts, and as they are posted (a listening connection is kept open). If a message is listed in the forum, then it is already in the local database.
Re: Measuring the page generation of the forum
On Friday, 13 April 2012 at 08:09:09 UTC, Vladimir Panteleev wrote: On Friday, 13 April 2012 at 03:24:19 UTC, Jesse Phillips wrote: I'm pretty sure it is the NG itself. Every news reader I've used has shown some form of "hiccup" If you try making a post under a heavy load you'll get a nice page telling you the load and what it needed to be. Long page loads are caused by a combination of high server load + RAM cache misses + SQLite queries that require a lot of seeks. Network operations, including fetching new posts from the NG and posting to the NG, are asynchronous, and never block web requests. I use the split-thread view, I haven't had long page loads so maybe I've miss understood what he means by a hiccup. But I have seen it take time to retrieve a message and display it to me. I would expect a page could be considered "loading" if it is still waiting on fetching the post from the NG.
Re: Measuring the page generation of the forum
On Friday, 13 April 2012 at 03:24:19 UTC, Jesse Phillips wrote: I'm pretty sure it is the NG itself. Every news reader I've used has shown some form of "hiccup" If you try making a post under a heavy load you'll get a nice page telling you the load and what it needed to be. Long page loads are caused by a combination of high server load + RAM cache misses + SQLite queries that require a lot of seeks. Network operations, including fetching new posts from the NG and posting to the NG, are asynchronous, and never block web requests.
Re: Measuring the page generation of the forum
On Friday, 13 April 2012 at 02:25:19 UTC, Somedude wrote: Well, it's not really urgent. It's fast enough. Still, collecting statistics over time would have been nice, so that one could analyse the general behaviour of the GC, for instance, by getting an idea of the standard deviation and maximum response time over weeks of use. Such statistics wouldn't be very meaningful due to the highly varying server load, caused by a large number of other websites and services running on the same machine. But I have experienced a response of several seconds once, I don't know where this comes from (although it's most likely one of your reasons). Long page loads are caused by a combination of high server load + RAM cache misses + SQLite queries that require a lot of seeks. I'm quite sure this is the case. BTW, could you repost the source code address ? Thx. There is a link to the source code on the Help page. Oh, I just saw you wrote the max page generation was around 50 ms. When there are no cache misses, yes.
Re: Measuring the page generation of the forum
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: Hi, I've noticed a couple of times (i.e very rarely, mind you), that the page generation of the forum had hiccups. Have other people experienced such hiccups ? If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault (or was it a network latency ?). Anyhow, the forum seems to me a good opportunity to measure the D runtime in action on the server side, and in particular the GC. A few questions : Is the server regularly restarted or is it running smoothly without having to be restarted at all ? Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ? I'm pretty sure it is the NG itself. Every news reader I've used has shown some form of "hiccup" If you try making a post under a heavy load you'll get a nice page telling you the load and what it needed to be.
Re: Measuring the page generation of the forum
Le 13/04/2012 03:40, Vladimir Panteleev a écrit : > On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: > There are some numbers in the reddit discussion: > > http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte > Oh, I just saw you wrote the max page generation was around 50 ms.
Re: Measuring the page generation of the forum
Le 13/04/2012 04:43, James Miller a écrit : > I doubt that the GC is the source of any real problems. As a web > developer (in PHP no less :S) I can say with authority that most slow > downs are due to waiting on external services and data processing. Some > of the slowest parts of our site are due to heavy database usage where > we have to perform over 100 SQL queries to generate a page. Other > slowdowns are caused by the need to process the data. > > I understand that you want to see how the GC affects the server > performance, but it will be so small that it will be lost in the noise, > a blip in traffic because it's lunchtime and people are watching youtube > in your area will cause a bigger delay than the GC kicking in. > > -- > James Miller I agree, you're probably right.
Re: Measuring the page generation of the forum
* Somedude [2012-04-13 04:25:19 +0200]: > Well, it's not really urgent. It's fast enough. > Still, collecting statistics over time would have been nice, so that one > could analyse the general behaviour of the GC, for instance, by getting > an idea of the standard deviation and maximum response time over weeks > of use. I guess since it can handle much larger loads, the forum server > is not stressed enough to give very meaningful insight under heavy load. > But I have experienced a response of several seconds once, I don't know > where this comes from (although it's most likely one of your reasons). > So collecting real time data would have to be done for each component of > the chain (SQLite, cache, D generated page). Such data would be useful > for any web server anyway. I doubt that the GC is the source of any real problems. As a web developer (in PHP no less :S) I can say with authority that most slow downs are due to waiting on external services and data processing. Some of the slowest parts of our site are due to heavy database usage where we have to perform over 100 SQL queries to generate a page. Other slowdowns are caused by the need to process the data. I understand that you want to see how the GC affects the server performance, but it will be so small that it will be lost in the noise, a blip in traffic because it's lunchtime and people are watching youtube in your area will cause a bigger delay than the GC kicking in. -- James Miller
Re: Measuring the page generation of the forum
Le 13/04/2012 03:40, Vladimir Panteleev a écrit : > On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: >> Hi, >> >> I've noticed a couple of times (i.e very rarely, mind you), that the >> page generation of the forum had hiccups. >> Have other people experienced such hiccups ? >> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's >> fault (or was it a network latency ?). > > Long page loads are caused by a combination of high server load + RAM > cache misses + SQLite queries that require a lot of seeks. I still > haven't found the time for the migration to a more powerful server. > Well, it's not really urgent. It's fast enough. Still, collecting statistics over time would have been nice, so that one could analyse the general behaviour of the GC, for instance, by getting an idea of the standard deviation and maximum response time over weeks of use. I guess since it can handle much larger loads, the forum server is not stressed enough to give very meaningful insight under heavy load. But I have experienced a response of several seconds once, I don't know where this comes from (although it's most likely one of your reasons). So collecting real time data would have to be done for each component of the chain (SQLite, cache, D generated page). Such data would be useful for any web server anyway. BTW, could you repost the source code address ? Thx. >> Is the server regularly restarted or is it running smoothly without >> having to be restarted at all ? > > It seems to run fine for weeks on end. Nice. > >> Does it measure the time for the page generation ? Would it be >> interesting to see at the bottom of the page something like "Page >> generated in xxx ms", when I (or other people) come accross those >> hiccups, and to collect these data for statistics over a week/month ? > > There are some numbers in the reddit discussion: > > http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte > Nice, now I remember having read this post.
Re: Measuring the page generation of the forum
Le 13/04/2012 01:55, Nick Sabalausky a écrit : > "SomeDude" wrote in message > news:mjectyruiwxebfrke...@forum.dlang.org... >> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: >> So noone thinks this could be a good idea ? > > I like it. (And the StopWatch in std.datetime would make it super easy. > StopWatch is freaking awesome.) But newgroup/forum posts don't always get > responses after only a few hours. And even a total lack of replies shouldn't > be mistaken for a "no" vote. So I wouldn't worry too much about that. > OK > Could you be more specific what you mean by "hiccup"? I assume you mean that > once in a while a page load will be slower than usual. If so, about how > long? > Yeah, once I was surprised, because it took several seconds I believe. > My wild random guesses, aside from your ideas, are that maybe your request > just happens to occur at the right time for the forum to decide to refresh > its cache (if it even works that way). Or maybe it could be related to the > newsgroup server (which the web forum is just a frontend for) being > temporarily down due to high load (which unfortunately has been happening a > lot lately). Vladimir would probably have a better idea what might be going > on than I would, though. > Ah, I forgot about the newsgroup server.
Re: Measuring the page generation of the forum
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: Hi, I've noticed a couple of times (i.e very rarely, mind you), that the page generation of the forum had hiccups. Have other people experienced such hiccups ? If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault (or was it a network latency ?). Long page loads are caused by a combination of high server load + RAM cache misses + SQLite queries that require a lot of seeks. I still haven't found the time for the migration to a more powerful server. Is the server regularly restarted or is it running smoothly without having to be restarted at all ? It seems to run fine for weeks on end. Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ? There are some numbers in the reddit discussion: http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
Re: Measuring the page generation of the forum
On Apr 12, 2012 4:29 PM, "SomeDude" wrote > > So noone thinks this could be a good idea ? I like it. I usually put a comment in the bottom of my pages but since the form is implemented in D it would nice to actually display it somewhere.
Re: Measuring the page generation of the forum
"SomeDude" wrote in message news:mjectyruiwxebfrke...@forum.dlang.org... > On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: >> Hi, >> >> I've noticed a couple of times (i.e very rarely, mind you), that the page >> generation of the forum had hiccups. >> Have other people experienced such hiccups ? >> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault >> (or was it a network latency ?). >> >> Anyhow, the forum seems to me a good opportunity to measure the D runtime >> in action on the server side, and in particular the GC. >> A few questions : >> Is the server regularly restarted or is it running smoothly without >> having to be restarted at all ? >> Does it measure the time for the page generation ? Would it be >> interesting to see at the bottom of the page something like "Page >> generated in xxx ms", when I (or other people) come accross those >> hiccups, and to collect these data for statistics over a week/month ? > > So noone thinks this could be a good idea ? I like it. (And the StopWatch in std.datetime would make it super easy. StopWatch is freaking awesome.) But newgroup/forum posts don't always get responses after only a few hours. And even a total lack of replies shouldn't be mistaken for a "no" vote. So I wouldn't worry too much about that. Could you be more specific what you mean by "hiccup"? I assume you mean that once in a while a page load will be slower than usual. If so, about how long? My wild random guesses, aside from your ideas, are that maybe your request just happens to occur at the right time for the forum to decide to refresh its cache (if it even works that way). Or maybe it could be related to the newsgroup server (which the web forum is just a frontend for) being temporarily down due to high load (which unfortunately has been happening a lot lately). Vladimir would probably have a better idea what might be going on than I would, though.
Re: Measuring the page generation of the forum
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: Hi, I've noticed a couple of times (i.e very rarely, mind you), that the page generation of the forum had hiccups. Have other people experienced such hiccups ? If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault (or was it a network latency ?). Anyhow, the forum seems to me a good opportunity to measure the D runtime in action on the server side, and in particular the GC. A few questions : Is the server regularly restarted or is it running smoothly without having to be restarted at all ? Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ? So noone thinks this could be a good idea ?
Measuring the page generation of the forum
Hi, I've noticed a couple of times (i.e very rarely, mind you), that the page generation of the forum had hiccups. Have other people experienced such hiccups ? If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault (or was it a network latency ?). Anyhow, the forum seems to me a good opportunity to measure the D runtime in action on the server side, and in particular the GC. A few questions : Is the server regularly restarted or is it running smoothly without having to be restarted at all ? Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ?
