> > Probably the test machine was changed, a new version of OpenSSL or
> pyOpenSSL, or something else?
>
> One of those things. There is no infrastructure in place for identifying
> events
> which impact the performance testing infrastructure. The only performance
Yes, this is an important point: track changes in infrastructure (everything
that might
have an influence, but is outside the tested code).
> testing environment is a very old mac mini still running Snow Leopard, which
omg;)
> > I'd say: the infrastructure aspects when doing performance tests do
> matter. To the degree that performance results are of very limited value at
> all, if the former aspects are not accounted for.
>
> I don't think the results that we have presently are worth much at all. My
> point was mostly that there is some infrastructure which is halfway usable,
> and so you don't have to start from scratch. If you could take over this
You mean taking over the code "as is"
http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/files
or the task in general (Twisted benchmarking)?
> project (I am pretty sure at this point there is nobody to take it over
> *from*,
We are currently developing performance test infrastructure for Crossbar.io -
naturally,
it is eating it's own dog food: the infrastructure is based on Crossbar.io and
WAMP to
orchestrate and wire up things in a distributed test setup.
We could extend that to test at the Twisted(-only) level. Need to think about
how
that fits into "overall strategy", as the Crossbar.io perf. test stuff isn't
open-source.
The testing hardware above (mac, no real network) is insufficient for what I
need.
I'm thinking about buying and setting up 2 more boxes for Linux.
Rgd. Codespeed (https://github.com/tobami/codespeed), which seems to be used
by speedcenter.twistedmatrix.com: I have issues here as well.
E.g. I need latency histograms, but this seems unsupported (benchmark results
can
only have avg/min/max/stddev). For me, this isn't "nice to have", but
essential.
Throughput is one thing. Constistent low latency a completely different. The
latter is
much much harder.
But what is the "interface" between test cases from "twisted-benchmarks" to
codespeed?
This
https://github.com/tobami/codespeed#saving-data
seems to suggest, performance test results are HTTP/POSTed as JSON to codespeed.
And codespeed is then only responsible for visualization and web hosting, right?
I think we can find something better for that part.
> (And if you care a lot about performance in a particular environment you
> could set it up in that environment and get attention for it :)).
Yes, in particular that very last one is a factor to justify efforts;) Anything
like having
a promo logo or similar - that would be an argument to invest time and material.
I will seriously contemplate .. need to align with strategy/available time.
We already host FreeBSD buildslaves for both Twisted and PyPy. That might be
another
synergy (hosting the latter on that same boxes).
> You should also have a look at the existing benchmark suite and potentially
> look at maintaining / expanding that as well.
I will try to integrate some of this into our upcoming perf. infrastructure.
/Tobias
>
> Thoughts?
>
> -glyph
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
