Re: HTTP server benchmarking/load testing in Python
On 1/26/2023 6:39 PM, Barry wrote: On 26 Jan 2023, at 17:32, Thomas Passin wrote: On 1/26/2023 11:41 AM, Chris Angelico wrote: On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: A nice theory but nothing to do with the real world. I've had a number of laptops that overheat (or would, if I let test program continue) running this test program. Define "overheat". If all you're saying is "the fan began to whine and I got annoyed so I shut off the program", that is absolutely NOT overheating. CPU core temperatures up to 95 deg C and rising rapidly, as reported by a number of utilities including NZXT and CoreTemp. Max junction temperature is given as 100 deg C, and I don't want to risk reducing the lifetime of my CPU. Silicon junctions melt something like 400C ish not 100C. The max you see is the operating temp of the CPU. Of course I know the junction isn't going to melt at 100 deg C. We're not talking low temperature solder here! For intel CPU if you go beyond what the slow clocking can deal with the CPU turns itself off to prevent damage. Intel did this to stop people asking for replacement parts when there cooling was at fault. Barry Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either. -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/26/2023 10:32 PM, Chris Angelico wrote: On Fri, 27 Jan 2023 at 14:21, Thomas Passin wrote: 2. "What is Tjunction max temperature?" Tjunction max is the maximum thermal junction temperature that a processor will allow prior to using internal thermal control mechanisms to reduce power and limit temperature. Activation of the processor's thermal control system may cause performance loss as the processor typically reduces frequency and power to prevent overheating. The maximum junction temperature limit varies per product and usually is between 100°C-110°C." https://www.intel.com/content/www/us/en/support/articles/05597/processors.html The utilities I used always stated a 100 deg limit for Tj. Yeah, so "maximum" is "before performance loss", not "before damage". Yeah, so a dozen years ago, when I first noticed the matter, most computers didn't have throttling and power reduction, so I got sensitized to it. Better safe than sorry. Airliners have safeguards against stalling, but it's a lot better not to try to stall them anyway (speaking as a (non-airline) pilot). Let's give this a rest, shall we? -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On Fri, 27 Jan 2023 at 14:21, Thomas Passin wrote: > 2. "What is Tjunction max temperature?" > Tjunction max is the maximum thermal junction temperature that a > processor will allow prior to using internal thermal control mechanisms > to reduce power and limit temperature. Activation of the processor's > thermal control system may cause performance loss as the processor > typically reduces frequency and power to prevent overheating. The > maximum junction temperature limit varies per product and usually is > between 100°C-110°C." > > https://www.intel.com/content/www/us/en/support/articles/05597/processors.html > > The utilities I used always stated a 100 deg limit for Tj. > Yeah, so "maximum" is "before performance loss", not "before damage". ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/26/2023 5:00 PM, Chris Angelico wrote: On Fri, 27 Jan 2023 at 06:54, Thomas Passin wrote: Did you get a warning, or did you just decide to stop the test? (At least) one of the utilities, I forget which one, did show the temperature in a danger zone. I'm very curious as to which utility, and on what basis it called it "danger". Notably, whether there's any sort of actual manufacturer threshold that that was based on. 1. we're talking maybe a dozen years ago, I don't remember every detail about wordings. Coretemp e.g., gives clear warnings (though at what I think are lower temperatures than necessary). 2. "What is Tjunction max temperature?" Tjunction max is the maximum thermal junction temperature that a processor will allow prior to using internal thermal control mechanisms to reduce power and limit temperature. Activation of the processor's thermal control system may cause performance loss as the processor typically reduces frequency and power to prevent overheating. The maximum junction temperature limit varies per product and usually is between 100°C-110°C." https://www.intel.com/content/www/us/en/support/articles/05597/processors.html The utilities I used always stated a 100 deg limit for Tj. 3. "Is it bad if my processor frequently approaches or reaches its maximum temperature? Not necessarily. Many Intel® processors make use of Intel® Turbo Boost Technology, which allows them to operate at very high frequency for a short amount of time. When the processor is operating at or near its maximum frequency it's possible for the temperature to climb very rapidly and quickly reach its maximum temperature. In sustained workloads, it's possible the processor will operate at or near its maximum temperature limit. Being at maximum temperature while running a workload isn't necessarily cause for concern. Intel processors constantly monitor their temperature and can very rapidly adjust their frequency and power consumption to prevent overheating and damage." (same source) But automatic throttling wasn't common back when I first noticed the heating issue. Personally? Very dubious. Your entire premise is "five degrees MUST be a problem", without any visible basis. Bridges are built with 150 - 200 % strength margin. This doesn't mean you should deliberately overload one. Heat is the enemy of electronics - a very old lesson. Tj =~ 100 deg C for CPUs, a familiar figure. My premise, to use your word, is not what you say. It is to avoid excessive heat if at all possible, and if the manufacturer says the max junction temperature is 100 deg, I'm going to avoid approaching 100 deg if possible - or to minimize the stay there. Most chemical effects are exponentially sensitive to temperature and problems with semiconductors are likely to be chemical - remember, e.g., the purple plague? A chemical problem. So yes, checking with HWiNFO, my current system is throttling and power limiting during this particular test. That's good. And I'm still going to stay away from the highest temperatures when possible. Nuff said! -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
> On 26 Jan 2023, at 17:32, Thomas Passin wrote: > > On 1/26/2023 11:41 AM, Chris Angelico wrote: >>> On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: >>> A nice theory but nothing to do with the real world. I've had a number >>> of laptops that overheat (or would, if I let test program continue) >>> running this test program. >> Define "overheat". If all you're saying is "the fan began to whine and >> I got annoyed so I shut off the program", that is absolutely NOT >> overheating. > > CPU core temperatures up to 95 deg C and rising rapidly, as reported by a > number of utilities including NZXT and CoreTemp. Max junction temperature is > given as 100 deg C, and I don't want to risk reducing the lifetime of my CPU. Silicon junctions melt something like 400C ish not 100C. The max you see is the operating temp of the CPU. For intel CPU if you go beyond what the slow clocking can deal with the CPU turns itself off to prevent damage. Intel did this to stop people asking for replacement parts when there cooling was at fault. Barry > > Maybe five or ten minutes at or above 100 deg C every few months might not > make a noticeable lifetime difference, who knows? I don't want to make a > habit of it. I wouldn't drive my car very long with a low oil pressure > warning active, either. > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On Fri, 27 Jan 2023 at 06:54, Thomas Passin wrote: > > Did you get a warning, or did you just decide to stop the test? > > (At least) one of the utilities, I forget which one, did show the > temperature in a danger zone. I'm very curious as to which utility, and on what basis it called it "danger". Notably, whether there's any sort of actual manufacturer threshold that that was based on. Personally? Very dubious. Your entire premise is "five degrees MUST be a problem", without any visible basis. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/26/2023 12:57 PM, Chris Angelico wrote: On Fri, 27 Jan 2023 at 04:31, Thomas Passin wrote: On 1/26/2023 11:41 AM, Chris Angelico wrote: On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: A nice theory but nothing to do with the real world. I've had a number of laptops that overheat (or would, if I let test program continue) running this test program. Define "overheat". If all you're saying is "the fan began to whine and I got annoyed so I shut off the program", that is absolutely NOT overheating. CPU core temperatures up to 95 deg C and rising rapidly, as reported by a number of utilities including NZXT and CoreTemp. Max junction temperature is given as 100 deg C, and I don't want to risk reducing the lifetime of my CPU. Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either. Did you get a warning, or did you just decide to stop the test? (At least) one of the utilities, I forget which one, did show the temperature in a danger zone. Did you continue the test and see what would happen? No, why would I? Would you go up to the edge of a cliff, past the warning signs, and when the ground started to crumble take another step to see if it would really collapse? Did you, when the temperature got up to 95°, check what the CPU's clock frequency was? The easiest way to recognize thermal throttling is a reduction in frequency while at 100% utilization. No, there was no point. Maybe it would have throttled, maybe no damage would have occurred. But doing so would not have accomplished anything, since I already had the throughput numbers I needed and the purpose of the test was not to see how hard I could drive the system before hardware failure. I'll leave that to Tom's Hardware or some gamers' site. Or did you just assume that, with a mere five degree buffer and your own personal analysis, that the CPU was just seconds away from total destruction? To quote myself from my last message: "Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either." -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On Fri, 27 Jan 2023 at 04:31, Thomas Passin wrote: > > On 1/26/2023 11:41 AM, Chris Angelico wrote: > > On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: > >> A nice theory but nothing to do with the real world. I've had a number > >> of laptops that overheat (or would, if I let test program continue) > >> running this test program. > > > > Define "overheat". If all you're saying is "the fan began to whine and > > I got annoyed so I shut off the program", that is absolutely NOT > > overheating. > > CPU core temperatures up to 95 deg C and rising rapidly, as reported by > a number of utilities including NZXT and CoreTemp. Max junction > temperature is given as 100 deg C, and I don't want to risk reducing the > lifetime of my CPU. > > Maybe five or ten minutes at or above 100 deg C every few months might > not make a noticeable lifetime difference, who knows? I don't want to > make a habit of it. I wouldn't drive my car very long with a low oil > pressure warning active, either. Did you get a warning, or did you just decide to stop the test? Did you continue the test and see what would happen? Did you, when the temperature got up to 95°, check what the CPU's clock frequency was? The easiest way to recognize thermal throttling is a reduction in frequency while at 100% utilization. Or did you just assume that, with a mere five degree buffer and your own personal analysis, that the CPU was just seconds away from total destruction? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/26/2023 11:41 AM, Chris Angelico wrote: On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: A nice theory but nothing to do with the real world. I've had a number of laptops that overheat (or would, if I let test program continue) running this test program. Define "overheat". If all you're saying is "the fan began to whine and I got annoyed so I shut off the program", that is absolutely NOT overheating. CPU core temperatures up to 95 deg C and rising rapidly, as reported by a number of utilities including NZXT and CoreTemp. Max junction temperature is given as 100 deg C, and I don't want to risk reducing the lifetime of my CPU. Maybe five or ten minutes at or above 100 deg C every few months might not make a noticeable lifetime difference, who knows? I don't want to make a habit of it. I wouldn't drive my car very long with a low oil pressure warning active, either. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 2023-01-26, Thomas Passin wrote: > On 1/26/2023 11:02 AM, Grant Edwards wrote: > >[...] > >> A properly designed laptop with a non-broken OS will not overheat >> regardless of the computing load you throw at it. The fan might get >> annoying loud, but if it overheats either your hardware or OS needs >> to be fixed. > > A nice theory but nothing to do with the real world. I've had a number > of laptops that overheat (or would, if I let test program continue) > running this test program. You mean they actually fail/crash? Or they just throttle the fans up and the CPU down to keep the core temperature within limits? -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On Fri, 27 Jan 2023 at 03:34, Thomas Passin wrote: > A nice theory but nothing to do with the real world. I've had a number > of laptops that overheat (or would, if I let test program continue) > running this test program. Define "overheat". If all you're saying is "the fan began to whine and I got annoyed so I shut off the program", that is absolutely NOT overheating. I would accept "the CPU thermally throttled to the point where the test was non-indicative" as a form of overheating, though then the warning should be "be aware that, if your web server is CPU-limited, this test may result in hard-to-interpret numbers due to requests per second varying with the change in CPU temperature", which isn't nearly as punchy. But unless you have a system where the heat sink isn't attached to the CPU properly, I'd be very surprised if you were able to actually damage your CPU this way. Maybe you could reduce the lifetime that way (the same way that crypto mining can shorten the lifespan of a GPU), but it shouldn't cause any sort of immediate damage. Even on a laptop. Feel free to prove me wrong, though. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/26/2023 11:02 AM, Grant Edwards wrote: On 2023-01-26, Thomas Passin wrote: On 1/25/2023 7:38 PM, Peter J. Holzer wrote: On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: Great! Don't forget what I said about potential overheating if you hit the server with as many requests as it can handle. Frankly, if you can overheat a server by hitting it with HTTP requests, get better hardware and/or put it into a place with better airflow. Frankly, if you have a server-grade machine then well and good but if you are running a nice quiet consumer grade laptop - my development machine - you need to be careful. A properly designed laptop with a non-broken OS will not overheat regardless of the computing load you throw at it. The fan might get annoying loud, but if it overheats either your hardware or OS needs to be fixed. A nice theory but nothing to do with the real world. I've had a number of laptops that overheat (or would, if I let test program continue) running this test program. They have been different brands, different CPUs, different levels of noisy fans. I don't know how I would find one of your "properly designed laptops with a non-broken OS", or what could be done to fix it. Maybe a high-end gaming machine... which I don't wish to invest in or hear the fan noise from. Anyway, the point was to warn other people - who probably also wouldn't have a "properly designed laptop with a non-broken OS" - that they should keep an eye on their CPU core temperatures. In my experience, that's a real concern, whether or not it "should not" be an issue. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 4:30 PM, Thomas Passin wrote: On 1/25/2023 3:29 PM, Dino wrote: Great! Don't forget what I said about potential overheating if you hit the server with as many requests as it can handle. Noted. Thank you. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
> On Jan 26, 2023, at 11:02 AM, Grant Edwards wrote: > > On 2023-01-26, Thomas Passin wrote: >> On 1/25/2023 7:38 PM, Peter J. Holzer wrote: >>> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: Great! Don't forget what I said about potential overheating if you hit the server with as many requests as it can handle. >>> >>> Frankly, if you can overheat a server by hitting it with HTTP requests, >>> get better hardware and/or put it into a place with better airflow. >>> >> >> Frankly, if you have a server-grade machine then well and good but if >> you are running a nice quiet consumer grade laptop - my development >> machine - you need to be careful. > > A properly designed laptop with a non-broken OS will not overheat > regardless of the computing load you throw at it. The fan might get > annoying loud, but if it overheats either your hardware or OS needs > to be fixed. Exactly. But what he might be thinking about is Thermal Throttling, which I keep seeing people attribute to overheating…. Overheating is not thermal throttling, it’s the OS and CPU protecting themselves from overheating. Usually because the manufacturer didn’t add enough cooling to keep the system cool enough with a continuous load. (Which to be honest, almost no laptop designers do, because they assuming you are going to be having a spiky load instead… - Benjamin -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 2023-01-26, Thomas Passin wrote: > On 1/25/2023 7:38 PM, Peter J. Holzer wrote: >> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: >>> Great! Don't forget what I said about potential overheating if you >>> hit the server with as many requests as it can handle. >> >> Frankly, if you can overheat a server by hitting it with HTTP requests, >> get better hardware and/or put it into a place with better airflow. >> > > Frankly, if you have a server-grade machine then well and good but if > you are running a nice quiet consumer grade laptop - my development > machine - you need to be careful. A properly designed laptop with a non-broken OS will not overheat regardless of the computing load you throw at it. The fan might get annoying loud, but if it overheats either your hardware or OS needs to be fixed. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 11:23 PM, Dino wrote: On 1/25/2023 3:27 PM, Dino wrote: On 1/25/2023 1:33 PM, orzodk wrote: I have used locust with success in the past. https://locust.io First impression, exactly what I need. Thank you Orzo! the more I learn about Locust and I tinker with it, the more I love it. Thanks again. That's the one I was trying to remember! I think it was in in its early days when I tried it out. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 3:27 PM, Dino wrote: On 1/25/2023 1:33 PM, orzodk wrote: I have used locust with success in the past. https://locust.io First impression, exactly what I need. Thank you Orzo! the more I learn about Locust and I tinker with it, the more I love it. Thanks again. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 8:36 PM, Chris Angelico wrote: On Thu, 26 Jan 2023 at 12:06, Thomas Passin wrote: On 1/25/2023 7:38 PM, Peter J. Holzer wrote: On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: Great! Don't forget what I said about potential overheating if you hit the server with as many requests as it can handle. Frankly, if you can overheat a server by hitting it with HTTP requests, get better hardware and/or put it into a place with better airflow. Frankly, if you have a server-grade machine then well and good but if you are running a nice quiet consumer grade laptop - my development machine - you need to be careful. We don't know what hardware the OP is using. And it's not servicing the requests per se that's the issue, it's the heavy computing load that has to be done for each request. The CPU is generally pegged at 100% for most or all of the test. If you have to worry about thermals because of CPU load, then worry about thermals because of CPU load. The HTTP request testing is completely separate. Load testing means putting a system under load. I'm not sure why you'd be concerned about one specific possible consequence, rather than, I dunno, just put the system under load and see how it performs? This is not that hard, folks! I needed to know the throughput of this system if it were hit with a great many queries at once, as if it were a busy help desk, for instance. The intent is not to bring the server to its knees as a load test for the server, it's to measure the maximum throughput for independent queries. It happens that each query takes a lot of processing, so the system is not IO bound, it's CPU bound. The result is a large CPU load and a large amount of heat generated. "just put the system under load and see how it performs" This is exactly what was happening, but not in the service of stressing the computer, but in finding the throughput for this particular Tomcat app with representative queries. As a byproduct, I noticed very high CPU temperatures, which probably wouldn't have occurred in a datacenter server with much better cooling. I didn't care about that except for protecting my own laptop. And this is way OT for the OP's question. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On Thu, 26 Jan 2023 at 12:06, Thomas Passin wrote: > > On 1/25/2023 7:38 PM, Peter J. Holzer wrote: > > On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: > >> Great! Don't forget what I said about potential overheating if you > >> hit the server with as many requests as it can handle. > > > > Frankly, if you can overheat a server by hitting it with HTTP requests, > > get better hardware and/or put it into a place with better airflow. > > > > Frankly, if you have a server-grade machine then well and good but if > you are running a nice quiet consumer grade laptop - my development > machine - you need to be careful. We don't know what hardware the OP is > using. And it's not servicing the requests per se that's the issue, > it's the heavy computing load that has to be done for each request. The > CPU is generally pegged at 100% for most or all of the test. If you have to worry about thermals because of CPU load, then worry about thermals because of CPU load. The HTTP request testing is completely separate. Load testing means putting a system under load. I'm not sure why you'd be concerned about one specific possible consequence, rather than, I dunno, just put the system under load and see how it performs? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 7:38 PM, Peter J. Holzer wrote: On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: Great! Don't forget what I said about potential overheating if you hit the server with as many requests as it can handle. Frankly, if you can overheat a server by hitting it with HTTP requests, get better hardware and/or put it into a place with better airflow. Frankly, if you have a server-grade machine then well and good but if you are running a nice quiet consumer grade laptop - my development machine - you need to be careful. We don't know what hardware the OP is using. And it's not servicing the requests per se that's the issue, it's the heavy computing load that has to be done for each request. The CPU is generally pegged at 100% for most or all of the test. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 2023-01-25 16:30:56 -0500, Thomas Passin wrote: > Great! Don't forget what I said about potential overheating if you > hit the server with as many requests as it can handle. Frankly, if you can overheat a server by hitting it with HTTP requests, get better hardware and/or put it into a place with better airflow. hp -- _ | Peter J. Holzer| Story must make more sense than reality. |_|_) || | | | h...@hjp.at |-- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" signature.asc Description: PGP signature -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 3:29 PM, Dino wrote: On 1/25/2023 1:21 PM, Thomas Passin wrote: I actually have a Python program that does exactly this. Thank you, Thomas. I'll check out Locust, mentioned by Orzodk, as it looks like a mature library that appears to do exactly what I was hoping. Great! Don't forget what I said about potential overheating if you hit the server with as many requests as it can handle. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 1:21 PM, Thomas Passin wrote: I actually have a Python program that does exactly this. Thank you, Thomas. I'll check out Locust, mentioned by Orzodk, as it looks like a mature library that appears to do exactly what I was hoping. -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 1:33 PM, orzodk wrote: I have used locust with success in the past. https://locust.io First impression, exactly what I need. Thank you Orzo! -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
Dino writes: > Hello, I could use something like Apache ab in Python ( > https://httpd.apache.org/docs/2.4/programs/ab.html ). > > The reason why ab doesn't quite cut it for me is that I need to define > a pool of HTTP requests and I want the tool to run those (as opposed > to running the same request over and over again) > > Does such a marvel exist? > > Thinking about it, it doesn't necessarily need to be Python, but I > guess I would have a chance to tweak things if it was. > > Thanks > > Dino I have used locust with success in the past. https://locust.io -- https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server benchmarking/load testing in Python
On 1/25/2023 10:53 AM, Dino wrote:
Hello, I could use something like Apache ab in Python (
https://httpd.apache.org/docs/2.4/programs/ab.html ).
The reason why ab doesn't quite cut it for me is that I need to define a
pool of HTTP requests and I want the tool to run those (as opposed to
running the same request over and over again)
Does such a marvel exist?
Thinking about it, it doesn't necessarily need to be Python, but I guess
I would have a chance to tweak things if it was.
I actually have a Python program that does exactly this. The intention
was to simulate a large number of independent users hitting a particular
web site as rapidly as possible, so see what the typical throughput is.
The program is somewhat specialized in the nature of the requests, but
the method is easy enough to implement.
The requests are composed from a pool of 300 pieces, and for each
request, four pieces are selected randomly with replacement and combined
to form the entire request. The idea here is to try to minimize caching,
so as to better assess the throughput for random queries.
The program runs a configurable number of threads. Each thread tries to
maintain an average query rate, but you have to throttle them to prevent
an exponential buildup of the request queue. If you run the program,
the server machine (usually the same as the querying machine) is likely
to get very hot - it's can be quite a stress test - and you want to
monitor the CPU temperatures just in case.
I can't share the actual code for copyright reasons, but the above
description should be helpful. The actual code is not very complicated
nor hard to develop.
I also have a version that uses async techniques instead of threads. To
give a feel for using a program like this, I think I can show the
'__main__' bit:
if __name__ == '__main__':
handleCmdLine()
# Warm up [redacted] in case it is not ready to get flooded with
queries
for n in range(WARMUP_REPS):
HTTPClient(HOST, setpath(), True)
asyncore.loop()
# Warmup done, reset hit counter
reps = 0
# And away we go ...
for n in range(NUMCLIENTS):
HTTPClient(HOST, setpath())
start = clock()
asyncore.loop(timeout=50)
now = clock()
sys.stderr.write('\n')
reps_per_sec = reps / (now - start)
print ('%0.1f hits/sec' % reps_per_sec)
There are also some polling-based systems available that do a similar
job. I don't remember the name of the one I tried a few years ago. It
ran a server to run the queries and reported the results via your browser.
--
https://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
On 5/4/2010 2:07 AM, Bryan wrote: The SQLite developers state the situation brilliantly at http://www.sqlite.org/whentouse.html: For future reference, that link does not work with Thunderbird. This one does. http://www.sqlite.org/whentouse.html When posting links, best to put them on a line by themselves, with NO punctuation, even if 'proper' English seems to require it. A space before the colon probably would have worked too, if its complete absence bothers one too much ;=). Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
On Mon, 3 May 2010 23:07:08 -0700 (PDT), Bryan wrote: >I love SQLite because it solves problems I actually have. For the vast >majority of code I write, "lite" is a good thing, and lite as it is, >SQLite can handle several transactions per second. I give SQLite a >file path and in a split second I have a relational, transactional >database. Great. I did not want to configure a server and I sure did >not want to inflict complexity upon my users. Exactly. I need a safe way to share an SQLite database among a few years, but sharing a drive isn't safe but I also don't need a full-fledged DBMS like MySQL. At this point, as an alternative to a Python-based solution, it seems like Mongoose + Lua + SQLite could do the trick. -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
John Nagle wrote: > [...] SQLite really > is a "lite" database. Although there's good read concurrency, multiple > updates from multiple processes tend to result in sizable delays, since > the locking is via file locks and wait/retry logic. True, and I have other gripes about SQLite, but I've fallen in love with it. SQLite rocks. SQLite rocks like Python rocks. Hard as Python had rocked before, Python started rockin' a whole bunch harder when 2.5 included SQLite3 in the standard library. I love SQLite because it solves problems I actually have. For the vast majority of code I write, "lite" is a good thing, and lite as it is, SQLite can handle several transactions per second. I give SQLite a file path and in a split second I have a relational, transactional database. Great. I did not want to configure a server and I sure did not want to inflict complexity upon my users. If you are smart and/or lucky enough to write a web app so popular that it outgrows SQLite, you can switch over to a big-time SQL server DBMS. At worst, you'll have to tweak some of the code. Imagine how much harder the scaling problem would be if the persistent data were stored via pickle. The SQLite developers state the situation brilliantly at http://www.sqlite.org/whentouse.html: "SQLite is not designed to replace Oracle. It is designed to replace fopen()." -- --Bryan -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
Gilles Ganault wrote: Hello I'd like to build a prototype that will combine a web server as front-end (it must support GZIPping data to the remote client when there are a lot of data to return), and SQLite as back-end, call the server from a VB.Net application, and see how well this works. I want to see if performance is significantly lower than using a server that uses a binary protocol. I'm no Python expert, so would appreciate any information on how to combine a web server and SQLite into a single Python application. This is just for a proof-of-concept, so it doesn't need to be shipping-quality. Thank you for any hint. There's no reason you can't do that. Whether you want to is another issue. There are, after all, plenty of web servers out there. Also, SQLite really is a "lite" database. Although there's good read concurrency, multiple updates from multiple processes tend to result in sizable delays, since the locking is via file locks and wait/retry logic. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
On May 3, 8:46 am, Gilles Ganault wrote: > Hello > > I'd like to build a prototype that will combine a web server as > front-end (it must support GZIPping data to the remote client when > there are a lot of data to return), and SQLite as back-end, call the > server from a VB.Net application, and see how well this works. I want > to see if performance is significantly lower than using a server that > uses a binary protocol. > > I'm no Python expert, so would appreciate any information on how to > combine a web server and SQLite into a single Python application. This > is just for a proof-of-concept, so it doesn't need to be > shipping-quality. > > Thank you for any hint. I quite like web.py: http://webpy.org/ L. -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
On Mon, 03 May 2010 11:51:41 +0200, Helmut Jarausch wrote: >http://www.karrigell.fr/doc/ Thanks for the tip. -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
interesting natalie5...@hotmail.co.jp natalie53...@yahoo.co.jp natalie5...@gmail.com wlipgf...@jupiter.ocn.ne.jp natalie5...@mbr.nifty.com natalie5...@xqg.biglobe.ne.jp natalie5...@zpost.plala.or.jp -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
On 05/03/10 09:46, Gilles Ganault wrote: > Hello > > I'd like to build a prototype that will combine a web server as > front-end (it must support GZIPping data to the remote client when > there are a lot of data to return), and SQLite as back-end, call the > server from a VB.Net application, and see how well this works. I want > to see if performance is significantly lower than using a server that > uses a binary protocol. > > I'm no Python expert, so would appreciate any information on how to > combine a web server and SQLite into a single Python application. This > is just for a proof-of-concept, so it doesn't need to be > shipping-quality. > You might have a look at http://www.karrigell.fr/doc/ Helmut. -- Helmut Jarausch Lehrstuhl fuer Numerische Mathematik RWTH - Aachen University D 52056 Aachen, Germany -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
On Mon, May 3, 2010 at 12:46 AM, Gilles Ganault wrote: > I'd like to build a prototype that will combine a web server as > front-end (it must support GZIPping data to the remote client when > there are a lot of data to return), and SQLite as back-end, call the > server from a VB.Net application, and see how well this works. I want > to see if performance is significantly lower than using a server that > uses a binary protocol. > > I'm no Python expert, so would appreciate any information on how to > combine a web server and SQLite into a single Python application. This > is just for a proof-of-concept, so it doesn't need to be > shipping-quality. If your want to write a basic/low-level HTTP server: http://docs.python.org/library/basehttpserver.html Looks like you'd use HTTPServer and a custom subclass of BaseHTTPRequestHandler. If you want to write at the slightly higher WSGI (http://www.python.org/dev/peps/pep-0333/) level of abstraction, you can have your WSGI application run by a simple Python HTTP server such as: http://pythonpaste.org/modules/httpserver.html As Alex said, SQLite is in the std lib: http://docs.python.org/library/sqlite3 Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server + SQLite?
Gilles Ganault wrote: > I'm no Python expert, so would appreciate any information on how to > combine a web server and SQLite into a single Python application. Hey Gilles, I'm a fan of the http framework, CherryPy[1]. Very quick and easy to get something up and running. The site also has some ideas on interoperating with a database[2], although this might be a better starting point[3]. SQLite is included in the Python standard library past 2.5, but if you're stuck with either 2.3 or 2.4, the 3rd party library pysqlite[4] provides support for them. For a higher level web framework, I find Turbogears 2.x[5] really straightforward. It's based around SQLAlchemy, which supports SQLite, but sounds like it's probably overkill for your situation. 1: http://www.cherrypy.org/ 2: http://tools.cherrypy.org/wiki/Databases 3: http://code.activestate.com/recipes/496799/ 4: http://code.google.com/p/pysqlite 5: http://turbogears.org/2.0/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Http server
Does anybody know how to redirect a post request ? i have a js file that does a post request to a /php/action.php file and i would like for the secretary to just do the action method instead that is defined in her python Http class book, so i can run both php and python without changing the static source code at 2 different ports head to head to see witch one can handle the most requests ? -- http://mail.python.org/mailman/listinfo/python-list
Re: Http server
> > The cute secretary's name is "cherrypy.tools.staticdir".
> > Check out her resume at http://www.cherrypy.org/wiki/StaticContent
>
> I think i am in love :)
Cant believe this just works out
import os.path
import cherrypy
pwd = os.path.dirname(os.path.abspath(__file__))
class Http:
_cp_config = {'tools.sessions.on': True}
@cherrypy.expose
def index(self):
f = open(os.path.join(pwd, '../htm/index.htm'))
xml = f.read()
f.close()
return xml
if __name__ == '__main__':
cherrypy.config.update({'server.socket_port': 8080,
'server.thread_pool': 10,
'environment': 'production',
'log.screen': True})
conf = {'/': {'tools.staticdir.root': os.path.join(pwd, '../htm')},
'/css': {'tools.staticdir.on': True,
'tools.staticdir.dir': os.path.join(pwd, '../css')},
'/js': {'tools.staticdir.on': True,
'tools.staticdir.dir': os.path.join(pwd, '../js')}}
cherrypy.quickstart(Http(), '/', config=conf)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Http server
> The cute secretary's name is "cherrypy.tools.staticdir". > Check out her resume at http://www.cherrypy.org/wiki/StaticContent I think i am in love :) -- http://mail.python.org/mailman/listinfo/python-list
Re: Http server
Gert Cuykens wrote: > so far this works > > > import cherrypy > import os.path > > class Http: > > def index(self): > f = open(os.path.join(os.path.dirname(__file__), '../htm/index.htm')) > xml = f.read() > f.close() > return xml > index.exposed = True > > cherrypy.tree.mount(Http()) > > if __name__ == '__main__': > cherrypy.config.update(os.path.join(os.path.dirname(__file__), > 'server.conf')) > cherrypy.server.quickstart() > cherrypy.engine.start() > > > I would like a cute secretary for Christmas but i dont think i will > get one. So i need to find a way to scan a directory. And map all > files inside the directory and generate a class member like > > > def filename(self): > f = open(os.path.join(os.path.dirname(__file__), '../htm/filename')) > xml = f.read() > f.close() > return xml > filename.exposed = True > > > Any idea or cute secretary maybe ? The cute secretary's name is "cherrypy.tools.staticdir". Check out her resume at http://www.cherrypy.org/wiki/StaticContent Robert Brewer System Architect Amor Ministries [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP Server Root Folder
placid wrote:
> Hi All,
>
> I have this BaseHTTPServer.HTTPServer that is located at C:\ (im on
> Windows XP), when i run the program (httpserver.pyw) from the Run
> Dialog as "C:\httpserver.pyw" the root folder ("\") for http server is
> C:\, but when i add an entry to Registry Run so that it runs at boot
> time, the root folder becomes "C:\Documents and Settings\" ?
>
> Does anyone know what the problem is?
>
Yes. Rather obviously, these two different ways of running programs
assert different working directories for the program.
A simple way around this would be to have your program make an
os.chdir() call to ensure the correct working directory before it starts
the server.
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden
--
http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server
Simon Forman wrote:
> ...
> >
>
> Awesome! Glad to hear it.
>
> ...
> >
> > Thanks for the help. I got it to work now.
> >
>
> You're welcome. I'm glad I could help you. :-D
>
Im having trouble with the following code for handling GET requests
from a client to my HTTP server. What i want to do is restrict access
only to a folder and contents within this folder. But when trying to
open files (text files) i get file not found error from send_head()
method of SimpleHTTPServer. The reason behind this is when opening the
file the path to the file is only C:\file.txt when it should be
C:\folder\file.txt. And when i remove the code that checks if path
contains "txt" it works (i can access files without errors).
Any help will be greatly appreciated!
def list_directory(self, path):
"""Helper to produce a directory listing (absent index.html).
Return value is either a file object, or None (indicating an
error). In either case, the headers are sent, making the
interface the same as for send_head().
"""
f = StringIO()
p = self.translate_path(self.path)
if p.find("txt") == -1:
f.write("httpserver.py: Access Denied" )
f.write("httpserver.py: Access Denied" )
else:
try:
list = os.listdir(path)
except os.error:
self.send_error(404, "No permission to list directory")
return None
list.sort(key=lambda a: a.lower())
displaypath = cgi.escape(urllib.unquote(self.path))
f.write("Directory listing for %s\n" %
displaypath)
f.write("Directory listing for %s\n" %
displaypath)
f.write("\n\n")
for name in list:
fullname = os.path.join(path, name)
displayname = linkname = name
# Append / for directories or @ for symbolic links
if os.path.isdir(fullname):
displayname = name + "/"
linkname = name + "/"
if os.path.islink(fullname):
displayname = name + "@"
# Note: a link to a directory displays with @ and
links with /
f.write('%s\n'
% (urllib.quote(linkname),
cgi.escape(displayname)))
f.write("\n\n")
length = f.tell()
f.seek(0)
self.send_response(200)
self.send_header("Content-type", "text/html")
self.send_header("Content-Length", str(length))
self.end_headers()
return f
--
http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server
placid wrote: > Simon Forman wrote: ... > > The file was named test.cgi. I changed it too test.py and it worked > Awesome! Glad to hear it. ... > > Thanks for the help. I got it to work now. > You're welcome. I'm glad I could help you. :-D Peace, ~Simon -- http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server
Simon Forman wrote:
> placid wrote:
> > Simon Forman wrote:
> > >
> ...
> > > For what you're asking about you'd probably want to use the
> > > CGIHTTPRequestHandler from the CGIHTTPServer module instead. Check out
> > > http://docs.python.org/lib/module-CGIHTTPServer.html
> >
> > This is what i was after, thanks for the tip.
> >
>
> You're welcome, my pleasure. : )
>
> ...
> >
> > Im having trouble running the following cgi script on windows
> >
> >
> >
> > #!c:/Python/python.exe -u
> >
> > text = """Content-type: text/html
> >
> > CGI 101
> > A Second CGI script
> >
> > Hello, CGI World!
> > """
> > print text
> >
> >
> >
> >
> > using this http server from
> > http://effbot.org/librarybook/cgihttpserver.htm
> >
> >
> >
> > import CGIHTTPServer
> > import BaseHTTPServer
> >
> > class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
> > cgi_directories = ["/cgi"]
> >
> > PORT = 8000
> >
> > httpd = BaseHTTPServer.HTTPServer(("", PORT), Handler)
> > print "serving at port", PORT
> > httpd.serve_forever()
> >
> >
> >
> > i get the error number 403, when i try to access the cgi script which
> > is located in a subdirectory called cgi where this file is (http
> > server). I have a feeling that i need to change the Handler class or
> > something, implement , but i couldnt find any examples other then this
> > from eff-bot.
> >
> > It could also be this line of code, that a google search turned up, its
> > the way of running cgi scripts on windows
> >
> > #!c:/Python/python.exe -u
> >
> >
> > Cheers
>
>
> Your cgi and server scripts look fine to me.
>
> Some diagnostic questions:
>
> What is the filename of your cgi script?
> (It must end in ".py" or ".pyw" for CGIHTTPRequestHandler to recognize
> it as a python script. See the is_python() method in the handler
> class. Also, in this case the "#!c:/Python/python.exe -u" line isn't
> needed, but it doesn't hurt. CGIHTTPRequestHandler should find the
> python interpreter for you without it.)
The file was named test.cgi. I changed it too test.py and it worked
>
> What was the exact URL that you're using to try to reach your script?
> (It should look something like:
> "http://localhost:8000/cgi/myscript.py";)
i was trying to access it at http://localhost:8000/test.cgi
which i now know is wrong.
>
> What was the error message that accompanied the 403 error?
>
> Did you start the server script from the directory it's in?
i started the server one directory above the cgi directory
> What was the log output from your server script when you tried to
> access the cgi script?
The error message outputted by the server was
localhost - - [26/Jun/2006 09:56:53] code 403, message CGI script is
not a plain file ('/cgi/')
Thanks for the help. I got it to work now.
Cheers
--
http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server
placid wrote:
> Simon Forman wrote:
> >
...
> > For what you're asking about you'd probably want to use the
> > CGIHTTPRequestHandler from the CGIHTTPServer module instead. Check out
> > http://docs.python.org/lib/module-CGIHTTPServer.html
>
> This is what i was after, thanks for the tip.
>
You're welcome, my pleasure. : )
...
>
> Im having trouble running the following cgi script on windows
>
>
>
> #!c:/Python/python.exe -u
>
> text = """Content-type: text/html
>
> CGI 101
> A Second CGI script
>
> Hello, CGI World!
> """
> print text
>
>
>
>
> using this http server from
> http://effbot.org/librarybook/cgihttpserver.htm
>
>
>
> import CGIHTTPServer
> import BaseHTTPServer
>
> class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
> cgi_directories = ["/cgi"]
>
> PORT = 8000
>
> httpd = BaseHTTPServer.HTTPServer(("", PORT), Handler)
> print "serving at port", PORT
> httpd.serve_forever()
>
>
>
> i get the error number 403, when i try to access the cgi script which
> is located in a subdirectory called cgi where this file is (http
> server). I have a feeling that i need to change the Handler class or
> something, implement , but i couldnt find any examples other then this
> from eff-bot.
>
> It could also be this line of code, that a google search turned up, its
> the way of running cgi scripts on windows
>
> #!c:/Python/python.exe -u
>
>
> Cheers
Your cgi and server scripts look fine to me.
Some diagnostic questions:
What is the filename of your cgi script?
(It must end in ".py" or ".pyw" for CGIHTTPRequestHandler to recognize
it as a python script. See the is_python() method in the handler
class. Also, in this case the "#!c:/Python/python.exe -u" line isn't
needed, but it doesn't hurt. CGIHTTPRequestHandler should find the
python interpreter for you without it.)
What was the exact URL that you're using to try to reach your script?
(It should look something like:
"http://localhost:8000/cgi/myscript.py";)
What was the error message that accompanied the 403 error?
Did you start the server script from the directory it's in?
What was the log output from your server script when you tried to
access the cgi script?
Peace,
~Simon
--
http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server
Simon Forman wrote:
>
>
> Ok, seriously, I don't know how pydoc does it, but when I need a
> quick-and-dirty http server [written in python] I use something like
> this:
>
> from BaseHTTPServer import HTTPServer
> from SimpleHTTPServer import SimpleHTTPRequestHandler
>
> HTTPServer(('', 8000), SimpleHTTPRequestHandler).serve_forever()
>
> For what you're asking about you'd probably want to use the
> CGIHTTPRequestHandler from the CGIHTTPServer module instead. Check out
> http://docs.python.org/lib/module-CGIHTTPServer.html
This is what i was after, thanks for the tip.
> Then you'd just write one or more cgi scripts (they can be in python
> IIRC) to run the commands you want.
Im having trouble running the following cgi script on windows
#!c:/Python/python.exe -u
text = """Content-type: text/html
CGI 101
A Second CGI script
Hello, CGI World!
"""
print text
using this http server from
http://effbot.org/librarybook/cgihttpserver.htm
import CGIHTTPServer
import BaseHTTPServer
class Handler(CGIHTTPServer.CGIHTTPRequestHandler):
cgi_directories = ["/cgi"]
PORT = 8000
httpd = BaseHTTPServer.HTTPServer(("", PORT), Handler)
print "serving at port", PORT
httpd.serve_forever()
i get the error number 403, when i try to access the cgi script which
is located in a subdirectory called cgi where this file is (http
server). I have a feeling that i need to change the Handler class or
something, implement , but i couldnt find any examples other then this
from eff-bot.
It could also be this line of code, that a google search turned up, its
the way of running cgi scripts on windows
#!c:/Python/python.exe -u
Cheers
--
http://mail.python.org/mailman/listinfo/python-list
Re: HTTP server
placid wrote:
> Hi all,
>
> Ive been reading about creating a HTTP server like the one pydoc
> creates (and studying pydoc source code). What i want to know, is it
> possible to create server that creates a webpage with hyperlinks that
> communicate back to the HTTP server, where each link accessed tells the
> server to execute some arbitrary command on local machine its running
> on?
>
> Cheers
Yes. It is possible.
Ok, seriously, I don't know how pydoc does it, but when I need a
quick-and-dirty http server [written in python] I use something like
this:
from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
HTTPServer(('', 8000), SimpleHTTPRequestHandler).serve_forever()
For what you're asking about you'd probably want to use the
CGIHTTPRequestHandler from the CGIHTTPServer module instead. Check out
http://docs.python.org/lib/module-CGIHTTPServer.html
Then you'd just write one or more cgi scripts (they can be in python
IIRC) to run the commands you want.
HTH,
~Simon
--
http://mail.python.org/mailman/listinfo/python-list
