Wow! Thanks for all of this work. It is really interesting to see how different the performance is on the Internet workload!
On Thu, Sep 7, 2017 at 9:52 PM, dbohdan <[email protected]> wrote: > On Tuesday, September 5, 2017 at 12:41:04 PM UTC+3, Jay McCarthy wrote: >> Is the benchmarking client core the same core as the server core? >> Could that help explain why single threaded performance is best? > > The not-quite-yes-or-no answer is that they were limited to separate virtual > cores inside a VirtualBox VM. When a VirtualBox VM has N virtual cores on a > physical CPU with M cores, it can use roughly up to N/M of the CPU's > resources. In that benchmark the applications, which had access to two > virtual cores, had to themselves 50% of the four-core physical CPU, and the > load generator had 25% with one virtual core. > > I've run three variants of the benchmark to see if running in a VM had a > noticeable effect on single-threaded vs. multi-threaded performance and to > address your question about whether "many" underperformed because of a > virtual network. > > First, I got rid of the VM. Both the applications and the load generator ran > in containers on the same machine, but not in a VM. This meant they were > limited to different physical cores. The results were similar to those in a > VM. The numbers were lower overall due to the slightly weaker hardware. > > ====== > results/caddy.txt:Requests per second: 2758.06 [#/sec] (mean) > results/compojure.txt:Requests per second: 2670.11 [#/sec] (mean) > results/custom-many-places.txt:Requests per second: 4326.27 [#/sec] (mean) > results/custom-many.txt:Requests per second: 4655.04 [#/sec] (mean) > results/custom-places.txt:Requests per second: 4584.75 [#/sec] (mean) > results/custom-single.txt:Requests per second: 5191.93 [#/sec] (mean) > results/flask.txt:Requests per second: 1111.25 [#/sec] (mean) > results/guile.txt:Requests per second: 1933.10 [#/sec] (mean) > results/plug.txt:Requests per second: 3346.99 [#/sec] (mean) > results/scgi.txt:Requests per second: 2092.03 [#/sec] (mean) > results/sinatra.txt:Requests per second: 293.60 [#/sec] (mean) > results/stateful.txt:Requests per second: 532.61 [#/sec] (mean) > results/stateless.txt:Requests per second: 625.02 [#/sec] (mean) > ====== > > Second, I ran the benchmark over a gigabit local network. Yesterday I pushed > a script for this (`remote-benchmark.exp`) to the repository. The > applications ran on one machine (in a Docker container with access to two > virtual cores). The load generator ran on another (my laptop). > > ====== > remote-results/caddy.txt-Requests per second: 3119.23 [#/sec] (mean) > remote-results/compojure.txt-Requests per second: 4009.71 [#/sec] (mean) > remote-results/custom-many-places.txt-Requests per second: 4409.48 [#/sec] > (mean) > remote-results/custom-many.txt-Requests per second: 5499.20 [#/sec] (mean) > remote-results/custom-places.txt-Requests per second: 5072.63 [#/sec] > (mean) > remote-results/custom-single.txt-Requests per second: 6246.09 [#/sec] > (mean) > remote-results/flask.txt-Requests per second: 1106.43 [#/sec] (mean) > remote-results/guile.txt-Requests per second: 2062.53 [#/sec] (mean) > remote-results/plug.txt-Requests per second: 4034.74 [#/sec] (mean) > remote-results/scgi.txt-Requests per second: 2046.91 [#/sec] (mean) > remote-results/sinatra.txt-Requests per second: 288.52 [#/sec] (mean) > remote-results/stateful.txt-Requests per second: 542.27 [#/sec] (mean) > remote-results/stateless.txt-Requests per second: 614.18 [#/sec] (mean) > ====== > > In both cases the ordering is still "single" > "many" > "places" > > "many-places". Though "many" and "places" are pretty close in the first case, > "many" consistently comes out ahead if you retest. > > Last, I ran the benchmark over the Internet with two machines about > 1.89×10^-10 light years apart. The applications ran on a very humble VPS. Due > to its humbleness I had to reduce the number of concurrent connections to 25. > "places", "many-places", and racket-scgi ran out of memory with as few as 10 > concurrent connections (racket-scgi seemingly due to nginx), so I decided to > exclude them rather than reduce the number of connections further. > > ====== >> env CONNECTIONS=25 ./remote-benchmark.exp vps > remote-results/caddy.txt:Requests per second: 231.37 [#/sec] (mean) > remote-results/compojure.txt:Requests per second: 242.41 [#/sec] (mean) > remote-results/custom-many.txt:Requests per second: 250.35 [#/sec] (mean) > remote-results/custom-single.txt:Requests per second: 255.21 [#/sec] (mean) > remote-results/flask.txt:Requests per second: 235.26 [#/sec] (mean) > remote-results/guile.txt:Requests per second: 242.38 [#/sec] (mean) > remote-results/plug.txt:Requests per second: 244.98 [#/sec] (mean) > remote-results/sinatra.txt:Requests per second: 239.78 [#/sec] (mean) > remote-results/stateful.txt:Requests per second: 239.60 [#/sec] (mean) > remote-results/stateless.txt:Requests per second: 238.71 [#/sec] (mean) > ====== > > -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- -=[ Jay McCarthy http://jeapostrophe.github.io ]=- -=[ Associate Professor PLT @ CS @ UMass Lowell ]=- -=[ Moses 1:33: And worlds without number have I created; ]=- -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
