Hi Beam Devs, currently PerfkitBenchmarker (a tool used to invoke performance tests) has two phases that run gradle commands:
- Pre-build phase: this is where all the beam repo is build. This phase is to prepare the necessary artifacts so that it doesn't happen when executing tests. - Actual test running phase. After all necessary code is built we run the test and measure it's execution time. The execution time is displayed on the PerfKit dashboard [1]. After the recent mvn - gradle migration we noticed that we are unable to "Pre build" the code[2]. Because one of the python related tasks fails, the whole "preBuild" phase fails silently and the actual building happens in the "test running" phase which increases the execution time (this is visible in the plots on the dashboard). This whole situation made me wonder about several things, and I'd like to ask you for opinions. I think: - we should skip all the python related tasks while building beam for java performance tests in PerfKit. Those are not needed anyway when we are running java tests. Is it possible to skip them in one go (eg. the same fashion we skip all checks using -xcheck option)? - the same goes for Python tests: we should skip all java related tasks when building beam for python performance tests in PerfKit. Note that this bullet is something to be developed in the future, as beam_PerformanceTests_Python job (the only Python Performance test job) is failing for 4 months now and seems abandoned. IMO it should be done when someone will bring the test back to life. For now the job should be disabled. - we should modify Perfkit so that when the prebuild phase fails for some reason, the test is not executed. Now we don't do this and the test execution time depends on whether "gradle integrationTest" command builds something or just runs the test. IMO when the command has to build anything the execution time should not be included in the Dashboards, because it's a false result. What do you think of all this? [1] https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688 [2] https://issues.apache.org/jira/browse/BEAM-4256 Best regards, Łukasz Gajowy
