Le samedi 10 mars 2018 à 12:57 +0100, Łukasz Gajowy a écrit : > > - Integration tests: AFAIK we only run the ones in examples module and only > > on demand. What about running all the IT > > (in > > particular IO IT) as a cron job on a daily basis with direct runner? Please > > note that it will require some always up > > backend infrastructure. > Running IOITs on Direct runner is fairly easy now - all the testing > infrastructure is there and the only thing needed > is switching the runner to Direct so this is nice and low effort. +1 > > @Kenneth: currently we spin up required databases using Kubernetes (postgres, > mongo on it's way on my branch). We also > added a hdfs cluster setup but no Jenkins tests are fired on it on regular > basis (yet). We also had some problems > running IOITs on Flink and Spark, see BEAM-3370 and BEAM-3371 so this area > may need some more development. > > > > - Performance tests: what about running Nexmark SMOKE test suite in batch > > and streaming modes with all the runners > > on a > > daily basis and store the running times in a RRD database (to see > > performance regressions)? Please note that not all > > the > > queries run in all the runners in all the modes right now. Also, we have > > some streaming pipelines termination issues > > (see https://issues.apache.org/jira/browse/BEAM-2847) > +1 too. Currently Performance Tests store results in BigQuery. Do you guys > think it's a good idea to store all the > tests results (Nexmark and IOIT) in one database (not separately)? Or maybe > think otherwise? I think that storing separately (at least separate tables) makes sense. Indeed, nexmark output to store is a 4 columns table: queryNum, executionTime, throughput (nb of events/s), and size of the output collection. Whereas IO IT output is more a status table (testName, status) Etienne > > > 2018-03-10 6:59 GMT+01:00 Jean-Baptiste Onofré > <[email protected]>> : > > Good ideas ! > >
> > > > Validates runner tests and Integration tests should be nightly executed. > > > > > > For the Performance tests, it's a great idea, but not sure daily basis is > > required. Maybe two times per week ? As these tests could be long, we > > should avoid to block executors that could impact our PR build and master > > build. Maybe we can add Jenkins executors dedicated to PerfTest. > > > > > > Regards > > > > JB > > > > > > On 09/03/2018 12:08, Etienne Chauchot wrote: > > > > > > > > Hi guys, > > > > > > > > > I was looking at the various jenkins jobs and I wanted to submit a > > > proposition: > > > > > > > > > - Validates runner tests: currently run at PostCommit for all the > > > runners. I think it is the quickest way to see > > > > > > regressions. So keep it that way > > > > > > > > > - Integration tests: AFAIK we only run the ones in examples module and > > > only on demand. What about running all the IT (in > > > > > > particular IO IT) as a cron job on a daily basis with direct runner? > > > Please note that it will require some always up > > > > > > backend infrastructure. > > > > > > > > > - Performance tests: what about running Nexmark SMOKE test suite in batch > > > and streaming modes with all the runners on a > > > > > > daily basis and store the running times in a RRD database (to see > > > performance regressions)? Please note that not all the > > > > > > queries run in all the runners in all the modes right now. Also, we have > > > some streaming pipelines termination issues > > > > > > (see https://issues.apache.org/jira/browse/BEAM-2847) > > > > > > > > > I know that Stephen Sisk use to work on these topics. I also talked to > > > guys from Polidea. But As I understood, they > > > > > > launch mainly integration tests on Dataflow runner. > > > > > > > > > WDYT? > > > > > > > > > Etienne > > > > > > > > > > > >
