Hi Andrew, Le mercredi 18 juillet 2018 à 10:08 -0700, Andrew Pilloud a écrit : > Hi Etienne, > I've been asking around and it sounds like we should be able to get a > dedicated Jenkins node for performance tests.
Cool ! > Another thing that might help is making the runs a few times longer. They are > currently running around 2 seconds each, > so the total time of the build probably exceeds testing. You mean increasing the size of the nexmark input collection? Currently it is set to 100 000 events and is narrowed down to 10 000 IIRC for some queries (4 and 10 IIRC) > Internally at Google we are running them with 2000x as many events on > Dataflow, but a job of that size won't even > complete on the Direct Runner. Yes probably not > I didn't see the query 3 issues, but now that you point it out it looks like > a bug to me too. It seems to me too, I'll open a ticket. Etienne > Andrew > On Wed, Jul 18, 2018 at 1:13 AM Etienne Chauchot <echauc...@apache.org> wrote: > > Hi Andrew, > > Yes I saw that, except dedicating jenkins nodes to nexmark, I see no other > > way. > > Also, did you see query 3 output size on direct runner? Should be a > > straight line and it is not, I'm wondering if > > there is a problem with sate and timers impl in direct runner. > > Etienne > > Le mardi 17 juillet 2018 à 11:38 -0700, Andrew Pilloud a écrit : > > > I'm noticing the graphs are really noisy. It looks like we are running > > > these on shared Jenkins executors, so our > > > perf tests are fighting with other builds for CPU. I've opened an issue > > > https://issues.apache.org/jira/browse/BEAM > > > -4804 and am wondering if anyone knows an easy fix to isolate these jobs. > > > Andrew > > > On Fri, Jul 13, 2018 at 2:39 AM Łukasz Gajowy <lgaj...@apache.org> wrote: > > > > @Etienne: Nice to see the graphs! :) > > > > > > > > @Ismael: Good idea, there's no document yet. I think we could create a > > > > small google doc with instructions on how > > > > to do this. > > > > > > > > pt., 13 lip 2018 o 10:46 Etienne Chauchot <echauc...@apache.org> > > > > napisał(a): > > > > > Hi, > > > > > @Andrew, this is because I did not find a way to set 2 scales on the > > > > > Y axis on the perfkit graphs. Indeed > > > > > numResults varies from 1 to 100 000 and runtimeSec is usually bellow > > > > > 10s. > > > > > Etienne > > > > > Le jeudi 12 juillet 2018 à 12:04 -0700, Andrew Pilloud a écrit : > > > > > > This is great, should make performance work much easier! I'm going > > > > > > to get the Beam SQL Nexmark jobs > > > > > > publishing as well. (Opened > > > > > > https://issues.apache.org/jira/browse/BEAM-4774 to track.) I might > > > > > > take on the > > > > > > Dataflow runner as well if no one else volunteers. > > > > > > > > > > > > I am curious as to why you have two separate graphs for runtime and > > > > > > count rather then graphing runtime/count > > > > > > to get the throughput rate for each run? Or should that be a third > > > > > > graph? Looks like it would just be a > > > > > > small tweak to the query in perfkit. > > > > > > Andrew > > > > > > On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada <pabl...@google.com> > > > > > > wrote: > > > > > > > This is really cool Etienne : ) thanks for working on this.Our of > > > > > > > curiosity, do you know how often the > > > > > > > tests run on each runner? > > > > > > > > > > > > > > Best > > > > > > > -P. > > > > > > > > > > > > > > On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau > > > > > > > <rmannibu...@gmail.com> wrote: > > > > > > > > Awesome Etienne, this is really important for the (user) > > > > > > > > community to have that visibility since it is > > > > > > > > one of the most important aspect of the Beam's quality, kudo! > > > > > > > > > > > > > > > > > > > > > > > > Romain Manni-Bucau > > > > > > > > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book > > > > > > > > > > > > > > > > Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré > > > > > > > > <j...@nanthrax.net> a écrit : > > > > > > > > > It's really great to have these dashboards and integration in > > > > > > > > > Jenkins ! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Etienne for driving this ! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/07/2018 15:13, Etienne Chauchot wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm glad to announce that the CI of Beam has much improved > > > > > > > > > > ! Indeed > > > > > > > > > > > > > > > > > > > Nexmark is now included in the perfkit dashboards. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At each commit on master, nexmark suites are run and plots > > > > > > > > > > are created > > > > > > > > > > > > > > > > > > > on the graphs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've created 2 kind of dashboards: > > > > > > > > > > > > > > > > > > > - one for performances (run times of the queries) > > > > > > > > > > > > > > > > > > > - one for the size of the output PCollection (which should > > > > > > > > > > be constant) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are dashboards for these runners: > > > > > > > > > > > > > > > > > > > - spark > > > > > > > > > > > > > > > > > > > - flink > > > > > > > > > > > > > > > > > > > - direct runner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Each dashboard contains: > > > > > > > > > > > > > > > > > > > - graphs in batch mode > > > > > > > > > > > > > > > > > > > - graphs in streaming mode > > > > > > > > > > > > > > > > > > > - graphs for the 13 queries. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That gives more than a hundred of graphs (my right finger > > > > > > > > > > hurts after so > > > > > > > > > > > > > > > > > > > many clics on the mouse :) ). It is detailed that much so > > > > > > > > > > that anyone > > > > > > > > > > > > > > > > > > > can focus on the area they have interest in. > > > > > > > > > > > > > > > > > > > Feel free to also create new dashboards with more > > > > > > > > > > aggregated data. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks to Lukasz and Cham for reviewing my PRs and showing > > > > > > > > > > how to use > > > > > > > > > > > > > > > > > > > perfkit dashboards. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dashboards are there: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424 > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384 > > > > > > > > > > > > > > > > > > > <https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712>https://apache-beam-test > > > > > > > > > ing.appspot.com/explore?dashboard=5138380291571712 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520 > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464 > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Enjoy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Etienne > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
