A minor nit: This definition of "speed up" is the correct/common use -
you increase the number of nodes and hope to see that system get
proportionally faster. E.g., 10x the iron makes your query run 10x
faster if you have "perfect speed up". However, this is not a
correct/common use of the term "scale up" - see the classic paper in
Comm. ACM by DeWitt and Gray on parallel database systems and their
performance. "Batch scale up" is when you increase both the data set
size AND the system size in tandem - and you hope to keep performance
flat. This means 10x the iron will run 10x the problem size in the same
time as 1x ran the 1x problem This is what it means to provide "perfect
batch scale up". There is also "transaction scale up" - this is when
you increase the number of concurrent queries and the amount of iron in
tandem - e.g., 10x the offered load and 10x the iron - and again, a
"perfect" result is that the bigger system handles the bigger workload
with the same performance as the 1x/1x case.
I suggest moving to the common "scale up" notion (and in this case, the
common "batch scale up" notion). So, as you increase problem size, also
increase cluster size, and look for performance to stay flat as you goal.
Cheers,
Mike
On 11/17/13 12:33 PM, Eldon Carman wrote:
The goal of the benchmark tests are to highlight the parallel aspects of
VXQuery. The tests need to show how VXQuery scales. In addition, other
queries may be added to highlight our specific speed improvements or where
improvements can still be made. At first we want to show how the system
works with parallel queries. We focus on three types of queries: filtering,
aggregation and nested loops (join).
For these three queries the following scaling tests will be completed:
scale up and speed up.
* Scale up keeps the number of nodes in the cluster constant and increases
the data set in each successive test.
* Speed up keeps the data set size constant and increases the number of
nodes processing the data in each successive test. (
http://en.wikipedia.org/wiki/Speedup)
Still working on the specific queries for our GHCN daily data, but you can
see the draft version here:
https://svn.apache.org/repos/asf/incubator/vxquery/trunk/vxquery/vxquery-benchmark/src/main/resources/noaa-ghcn-daily/queries/
