nickva commented on issue #5801: URL: https://github.com/apache/couchdb/issues/5801#issuecomment-3609902997
> Firstly, the overhead of generating the random bytes and (some of) the benchmarking logic is included in the benchmark. This was mentioned in the Readme. > > Now, the overhead itself is fine if it is constant - the difference in bytes per second is the speed difference. However, `:crypto:rand` takes longer in an increase in `N`. Even if `N` was constant, we don't know if the _deviation_ in the time it takes is significant. This is the second flaw. Yeah good point, that should taken out of the timing loop. Pre-generate a few and then select between them or just go through some length ranges individually (benchmark with length 10, 100, 1000, 10000). Or if we stick with random go use a common seed value. > Finally, we're benchmarking the two implementations with two random different inputs. Is this still an apples to apples comparison? Maybe a better approach would be to benchmark with a common random input? > > I'm not sure if my benchmark suffers from some of the same flaws, but maybe moving from handwritten benchmarks to some benchmarking framework would be better? Well the benchmarking script is already there so we just updated it to use the latest built-in functions. But something like benchee or even better erlperf is nice, too. > Are you getting different results in this range? Typically how large are the bytes you encode? Yeah the ranges I am looking are 100 - 10000. A place b64 encoding would matter is when emitting changes feeds sequences. For Q=64, N=1 db the size of sequence string is about 1200 bytes. For Q=2 would it closer to 120 bytes and such. This also used for attachment encoding when they are in-line in the document body (as opposed to updated directly). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
