Bob Palowoda wrote:
Well, just one "thought" actually ...I think we need a smaller batch size for the "connection" case. Connections are very expensive, so I think a batch size of 10 (instead of 256) would probably suffice.Well yes but isn't that kind of hiding the anomaly?
I'm not saying we should ignore the anomaly. No, it seems serious enough to warrant some investigation. But I do think 256 is probably too big for general purpose testing in this case.
We only use batches to amortise loop/call overheads from the timings, and to improve the timing resolution. There is a trade-off between the batch size, the number of samples and the run duration. The batch size is also sometimes directly related to a consumable resource (such as initialised connections etc).
In 0.3.0, Bart introduced a new duration control model which is driven by the number of samples collected (one batch = one sample). Before this (in 0.2.19) we could only control the batch size and the benchmark duration (in seconds). This meant we ran a lot of tests far longer than necessary for a statistically significant result, and we paid the price in terms of the overall "bench" script duration.
I think it's fair to say that although we've made some good steps in the right direction with 0.3.0, there is still some tuning left to do. However, the change in from 0.2.19 to 0.3.0 has (unintentially) exposed some important anomolies.
For instance, I'm currently investigating some of the "pipe" cases, bound to a single CPU in "mp" mode. Sometimes the batch size is as low as 2 (compared to 1000 in 0.2.19), and this has exposed regular spikes which were previously amortised away.
Whenever libMicro chucks up a serious anomoly it needs investigating. But such activity needs to be orthogonal to improving libMicro's usefulness in general.
Philp.s. by the way, I will be adding an extra round trip outside benchmark() in some of the pipe cases in order to prime the helper processes ... however, this doesn't remove the anomoly I'm investigating.
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ perf-discuss mailing list [email protected]
