Hi Etienne, it is already possible, provided that there is appropriate Jenkins job defined (see examples here: [1],[2]). Either the reviewer or the author can run the seed job to load job definitions (by typing "Run seed job" in comment) and then run the test he/she is interested to run (by specifying the correct phrase in the GitHub comment, eg. "Run Java JdbcIO Performance Test". The results are then available on Jenkins so those are public too.
Regarding the infrastructure: currently, if a test requires any Kubernetes' infrastructure, it is set up by PerfKitBenchmarker tool before the test is actually run. After the test execution, all the infrastructure is torn down. This also is made automatically provided that all necessary Kubernetes' scripts are there. Despite the fact that it is possible, I must say that all the "Performance Testing Framework" needs improvement in the following areas (so should be considered as an ongoing work in progress): - documentation and instructions for the community (this is getting more urgent!) - support for other runners (currently only direct and Dataflow are supported, as there were some issues when we tried to integrate it with Spark and Flink) - support for other filesystems (currently only local and HDFS are supported) - rename and reorganize IT jobs in Jenkins (see: [3]) Also, I think it's worthy to look improvement in terms of job definitions (seed jobs overwrite all jobs so this can collide with other developers work). See the thread I started a while ago in [4] for further info. Best regards, Łukasz Gajowy [1] https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy [2] https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT.groovy [3] https://issues.apache.org/jira/browse/BEAM-4298 [4] https://lists.apache.org/thread.html/b1aaea2c7eadc7ca1d1326b94a8c4c3a67befc0753897fd7fa4a3a4e@%3Cdev.beam.apache.org%3E 2018-05-30 10:14 GMT+02:00 Etienne Chauchot <echauc...@apache.org>: > Hi guys > Part of the CI improvement work, I would suggest to enable running the > integration tests of the IOs from the github PR. > > Indeed, when doing a review, either the reviewer or the author needs to > run the IT. The problem is that the results are private. It would be good > to be able to run IT using a phrase in github (like the validates runner > tests) to have the results public like any other test in the PR. > But it would require the backend IT infrastructures (kubernates/docker > ...) to be always up and also to set their credentials/location in the > related jenkins groovy script. > > I opened: > https://issues.apache.org/jira/browse/BEAM-4427 > > Thoughts? > > Best > Etienne >
