>> Sometimes this group reminds me of a certain large company I worked for.
>> If they didn't have a solution addressing a problem, they'd pretend it
>> didn't matter and belittle anyone who questioned that version of reality.
>>
>
> That's not strictly true; what's happening here is that someone's
> published a cool-looking bar graph but nobody knows what it really
> means. I don't feel the need to delve into everyone's benchmarks to
> explain why Python is purportedly worse. If someone uses those kinds
> of numbers to decide which programming language to use, they have
> bigger problems.
If you dig a bit, the benchmark is scary…
As in stupid-scary.
It consists of, 7 tests, and then a composite score is generated:
JSON Serialization - In this test, each response is a JSON serialization of a
freshly-instantiated object that maps the key message to the value Hello, World!
Single Query - In this test, each request is processed by fetching a single row
from a simple database table. That row is then serialized as a JSON response.
Multiple Queries - In this test, each request is processed by fetching multiple
rows from a simple database table and serializing these rows as a JSON
response. The test is run multiple times: testing 1, 5, 10, 15, and 20 queries
per request. All tests are run at 512 concurrency.
Cached Queries - In this test, each request is processed by fetching multiple
cached objects from an in-memory database (the cache having been populated from
a database table either as needed or prior to testing) and serializing these
objects as a JSON response. The test is run multiple times: testing 1, 5, 10,
15, and 20 cached object fetches per request. All tests are run at 512
concurrency. Conceptually, this is similar to the multiple-queries test except
that it uses a caching layer.
Fortunes - In this test, the framework's ORM is used to fetch all rows from a
database table containing an unknown number of Unix fortune cookie messages
(the table has 12 rows, but the code cannot have foreknowledge of the table's
size). An additional fortune cookie message is inserted into the list at
runtime and then the list is sorted by the message text. Finally, the list is
delivered to the client using a server-side HTML template. The message text
must be considered untrusted and properly escaped and the UTF-8 fortune
messages must be rendered properly. Whitespace is optional and may comply with
the framework's best practices.
data Updates - This test exercises database writes. Each request is processed
by fetching multiple rows from a simple database table, converting the rows to
in-memory objects, modifying one attribute of each object in memory, updating
each associated row in the database individually, and then serializing the list
of objects as a JSON response. The test is run multiple times: testing 1, 5,
10, 15, and 20 updates per request. Note that the number of statements per
request is twice the number of updates since each update is paired with one
query to fetch the object. All tests are run at 512 concurrency. The response
is analogous to the multiple-query test.
plain text - In this test, the framework responds with the simplest of
responses: a "Hello, World" message rendered as plain text. The size of the
response is kept small so that gigabit Ethernet is not the limiting factor for
all implementations. HTTP pipelining is enabled and higher client-side
concurrency levels are used for this test (see the "Data table" view).
Here, I instead benchmark my django gallery app, using Apache Bench, and so
forth. I guess I’ve been over-achieving…
I have to admit, though, that these benchmarks certainly allow everyone to
play.
431 cherrypy 587 0.0%(0.0%)
Even cherrypy with it’s 587 per second replies with plain-text.
The tasks seem deceptively (?) simple?
But looking closer, the data table for each task, gives more details. For
example the plain text is run 4 different times, at 4 different client-side
concurrency levels are used… But the levels are: 256, 1024, 4096, and 16384.
That can’t be the concurrency/thread count??!?!?!?? I can believe 1,000 -
3,000, outrageously high, but believable. But 16K worth of
concurrency/threads? I doubt that Wikipedia even has to dial it that high?
I have to give them points for providing API latency, and framework overhead….
- Benjamin
--
https://mail.python.org/mailman/listinfo/python-list
