+1 for removing tests that are not maintained. Are there features in Perfkit that we would like to be using that we aren't? Can we make the integration with Perfkit less brittle?
If we aren't getting much and don't plan to get much value in the short term, removal makes sense to me. On Thu, Jun 27, 2019 at 3:16 AM Łukasz Gajowy <lgaj...@apache.org> wrote: > Hi all, > > moving the discussion to the dev list: > https://github.com/apache/beam/pull/8919. I think that Perfkit > Benchmarker should be removed from all our tests. > > Problems that we face currently: > > 1. Changes to Gradle tasks/build configuration in the Beam codebase > have to be reflected in Perfkit code. This required PRs to Perfkit which > can last and the tests break due to this sometimes (no change in Perfkit + > change already there in beam = incompatibility). This is what happened in > PR 8919 (above), > 2. Can't run in Python3 (depends on python 2 only library like > functools32), > 3. Black box testing which hard to collect pipeline related metrics, > 4. Measurement of run time is inaccurate, > 5. It offers relatively small elasticity in comparison with eg. > Jenkins tasks in terms of setting up the testing infrastructure (runners, > databases). For example, if we'd like to setup Flink runner, and reuse it > in consequent tests in one go, that would be impossible. We can easily do > this in Jenkins. > > Tests that use Perfkit: > > 1. IO integration tests, > 2. Python performance tests, > 3. beam_PerformanceTests_Dataflow (disabled), > 4. beam_PerformanceTests_Spark (failing constantly - looks not > maintained). > > From the IOIT perspective (1), only the code that setups/tears down > Kubernetes resources is useful right now but these parts can be easily > implemented in Jenkins/Gradle code. That would make Perfkit obsolete in > IOIT because we already collect metrics using Metrics API and store them in > BigQuery directly. > > As for point 2: I have no knowledge of how complex the task would be (help > needed). > > Regarding 3, 4: Those tests seem to be not maintained - should we remove > them? > > Opinions? > > Thank you, > Łukasz > > > > >
