Regarding the JDK version, we're already on JDK17 and I wouldn't recommend downgrading. The main question for me would be what's more important to the project: a) keeping a direct comparison with akka or b) showing a good overall score. If a) and b) then we'd need to apply code changes to both pekko and akka submissions. One example showing how important it is to understand the benchmark physics is the akka-http-slick-postgres submission, which even for a single-query per request is performing 50% max RPS of the blocking implementation in akka-http, and is failing the updates test when slick starts rejecting tasks based on max queue length (*).
Finally, some frameworks actually use these benchmarks to find and dissolve bottlenecks in hotspots of their code, see Netty in the latest blog post which btw also features pekko as new submission (**). Oliver (*) https://tfb-status.techempower.com/unzip/results.2023-11-12-22-36-41-240.zip/results/20231106203152/akka-http-slick-postgres/run/akka-http-slick-postgres.log (**) https://www.techempower.com/blog/2023/11/15/framework-benchmarks-round-22/ Am Sa., 18. Nov. 2023 um 10:12 Uhr schrieb Matthew de Detrich <[email protected]>: > I am not ontop of this too much, but one major feature that is planned to > be added specifically in Pekko 1.1.x release (and also in Pekko modules > 1.1.x) is enabling the Scala 2 inliner which should provide at least some > performance improvement (improvements would be more noticeable on early > JDK's, i.e. 8,11) so maybe it's a good idea to have a separate benchmarks > specifically for the 1.1.x series? > > On Tue, Nov 14, 2023 at 9:52 PM Oliver Trosien <[email protected]> > wrote: > > > Hi, > > > > another quick update regarding the TechEmpower benchmark(eting) > > contribution. While the final blog post is yet to come, the benchmark > data > > is already available (1), and overall pekko-http got a decent overall > > score, ending up somewhere in the middle of the field (slightly above > Akka > > of course :). The screenshot attached shows maximum RPS for the following > > test types (2): JSON: 197k, 1-query: 141k, 20-query: 10.5k, Fortunes: > 96k, > > Updates: 1.8k and Plaintext: 2.6M. > > > > (1) > > > https://www.techempower.com/benchmarks/#hw=ph&test=composite§ion=data-r22 > > (2) > > > https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview > > > > If we wanted to improve the score, there's for sure some low-hanging > > fruit, including fixing unreasonable parallelism on the DB updates test > > (multiple parallel update queries for each request without any > > transactional context) which I intentionally didn't touch because my > > primary goal was a direct comparison to the Akka implementation. After > that > > I would kindly ask the project to take over the annual polishing, > including > > upgrades and fixes. > > > > Cheers, > > Oliver > > > > > > > > Am Mo., 4. Sept. 2023 um 08:46 Uhr schrieb Oliver Trosien < > > [email protected]>: > > > >> Hi, > >> > >> quick update: pekko-http got merged into the test suite (1), and a first > >> preliminary test round (2) shows it's on par with akka-http (as > expected - > >> we didn't change the logic of the code itself). I'll post another update > >> when the final numbers are ready. > >> > >> Oliver > >> > >> (1) > >> > https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Scala/pekko-http > >> (2) > >> > https://www.techempower.com/benchmarks/#section=test&runid=074e8a70-d6fb-4f10-82f3-43e57c0965b5&test=json&l=zik0zj-35b&hw=ph&p=zik0z3-zik0zj-zik0zj-zik0zj-v2qiv3-1 > >> > >> Am Mi., 23. Aug. 2023 um 14:57 Uhr schrieb PJ Fanning < > >> [email protected]>: > >> > >>> I think we should be able to keep track of this in future. > >>> > >>> On Wed, 23 Aug 2023 at 13:20, Oliver Trosien <[email protected]> > >>> wrote: > >>> > > >>> > Hi PJ, > >>> > > >>> > sure, I can do the initial submission. My main point is that it only > >>> makes > >>> > sense if the project is willing to take over the long-term > >>> maintainence. > >>> > > >>> > Oliver > >>> > > >>> > > >>> > Am Mi., 23. Aug. 2023 um 11:15 Uhr schrieb PJ Fanning < > >>> [email protected] > >>> > >: > >>> > > >>> > > Hi Olivier, > >>> > > > >>> > > Thanks for sharing your code. Are you planning on submitting that > to > >>> > > TechEmpower/FrameworkBenchmarks ? > >>> > > > >>> > > On Wed, 23 Aug 2023 at 07:13, Oliver Trosien <[email protected] > > > >>> wrote: > >>> > > > > >>> > > > Hi, > >>> > > > > >>> > > > fair point. I have working code here (just the plain variant, not > >>> the one > >>> > > > with slick from Lightbend), > >>> > > > which is as you pointed out basically changing the imports plus > >>> doing a > >>> > > bit > >>> > > > of cleanup: > >>> > > > https://github.com/otrosien/FrameworkBenchmarks/pull/1/ > >>> > > > > >>> > > > Main difference to the akka version is that it runs on JDK17. > Feel > >>> free > >>> > > to > >>> > > > use this as a baseline. > >>> > > > > >>> > > > Regards, > >>> > > > Oliver > >>> > > > > >>> > > > > >>> > > > Am Di., 22. Aug. 2023 um 19:31 Uhr schrieb PJ Fanning < > >>> > > [email protected] > >>> > > > >: > >>> > > > > >>> > > > > Thanks Olivier. We have Pekko and Pekko HTTP releases that > could > >>> be > >>> > > > > benchmarked. The Akka code could probably be pretty easily > >>> forked and > >>> > > > > used to create Pekko equivalents (mainly just changing the > >>> imports and > >>> > > > > possibly some config names). > >>> > > > > > >>> > > > > I wouldn't expect there to be much difference between the Akka > >>> and > >>> > > > > Pekko performance numbers. > >>> > > > > > >>> > > > > If someone has time to do the initial setup, it should be > fairly > >>> > > > > straightforward for us to maintain it going forward. > >>> > > > > > >>> > > > > For my part, I'm more interested in getting all the Pekko > modules > >>> > > > > released and helping to get widely used Akka ecosystem libs to > >>> support > >>> > > > > Pekko too (or to create Pekko based forks). > >>> > > > > > >>> > > > > On Tue, 22 Aug 2023 at 18:10, Oliver Trosien < > >>> [email protected]> > >>> > > wrote: > >>> > > > > > > >>> > > > > > Hi, > >>> > > > > > > >>> > > > > > I wanted to ask the project team if they want to maintain an > >>> > > > > implementation > >>> > > > > > of the framework benchmark run for pekko (you can see the > akka > >>> > > > > > implementation here: (1)) > >>> > > > > > > >>> > > > > > I would totally understand if you'd object this kind of > >>> > > "Benchmarketing", > >>> > > > > > but it still would give the project some additional > >>> visibility. I do > >>> > > > > have a > >>> > > > > > local version which basically is a copy of the akka one. > While > >>> there > >>> > > are > >>> > > > > > some framework submissions driven by individuals, they > usually > >>> don't > >>> > > have > >>> > > > > > insights or long-term interest in maintaining these > >>> submissions and > >>> > > as a > >>> > > > > > result performance regressions can shed bad light on the > >>> project (2). > >>> > > > > > > >>> > > > > > They want to close submissions to the next round soon (3) - > >>> and it > >>> > > > > usually > >>> > > > > > only happens once a year, so we would need to be quick to get > >>> > > included. > >>> > > > > > WDYT? > >>> > > > > > > >>> > > > > > Regards, > >>> > > > > > Oliver > >>> > > > > > > >>> > > > > > (1) > >>> > > > > > > >>> > > > > > >>> > > > >>> > https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Scala/akka-http > >>> > > > > > (2) > >>> > > > > > > >>> > > > > > >>> > > > >>> > https://github.com/TechEmpower/FrameworkBenchmarks/issues/7475#issuecomment-1607479108 > >>> > > > > > (3) > >>> > > > > > > >>> > > > > > >>> > > > >>> > https://github.com/TechEmpower/FrameworkBenchmarks/issues/7475#issuecomment-1684087157 > >>> > > > > > >>> > > > > > >>> --------------------------------------------------------------------- > >>> > > > > To unsubscribe, e-mail: [email protected] > >>> > > > > For additional commands, e-mail: [email protected] > >>> > > > > > >>> > > > > > >>> > > > >>> > > > --------------------------------------------------------------------- > >>> > > To unsubscribe, e-mail: [email protected] > >>> > > For additional commands, e-mail: [email protected] > >>> > > > >>> > > > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >>> > >>> > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > -- > > Matthew de Detrich > > *Aiven Deutschland GmbH* > > Immanuelkirchstraße 26, 10405 Berlin > > Alexanderufer 3-7, 10117 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > *m:* +491603708037 > > *w:* aiven.io *e:* [email protected] >
