Joshua Chamas wrote:
> There is no way that people are going to benchmark
> 10+ different environments themselves, so this merely offers
> a quick fix to get people going with their own comparisons.
I agree that having the code snippets for running hello world on
different tools collected in one place is handy.
> Do you have any idea how much time it takes to do these?
Yes, I've done quite a few of them. I never said they were easy.
> In order to improve the benchmarks, like the Resin & Velocigen
> ones that you cited where we have a very small sample, we simply
> need more numbers from more people.
I think we would need more numbers from the exact same people, on the
same machines, with the same configuration, the same client, the same
network, the same Linux kernel... In other words, controlled conditions.
> Also, any disclaimer modifications might be good if you feel
> there can be more work done there.
Ideally, I would get rid of every page except the one which lists the
tests grouped by OS/machine. Then I would put a big statement at the
top saying that comparisons across different people's tests are
meaningless.
- Perrin
