When do you mean by separate package? Most of our perf tests are integrated with PerfkitBenchmarker such as our JDBC benchmark[1] and Python BigQuery benchmark[2] and point to a specific IT class/method that contains the benchmarks.
Note that there was a recent discussion about moving away from PerfkitBenchmarker though and it would be good to follow up with Lukasz Gajowy about it since he has done a bunch of the existing benchmarks[3]. Finally, there were some docs about benchmarking on our design docs[4] page that may provide additional details[5, 6]. 1: https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy 2: https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy 3: https://lists.apache.org/thread.html/dab1c093799248787e8b75e63b66d7389b594b649a4d9a4a5db1cfbb@%3Cdev.beam.apache.org%3E 4: https://beam.apache.org/contribute/design-documents/ 5: https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE 6: https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA On Mon, Jul 15, 2019 at 8:03 AM Michał Walenia <michal.wale...@polidea.com> wrote: > Thanks for the information. I don't think that the task you mentioned fits > my case - I want to run a performance test on a real service and running it > with a battery of other tests doesn't make much sense to me. > It's similar to the case of other integration tests that are excluded from > the task you linked. > Is keeping the test in a separate package viable in your opinion? > > Thanks! > Michal > > On Fri, Jul 12, 2019 at 3:45 PM Lukasz Cwik <lc...@google.com> wrote: > >> Yes, there is a dependency between Dataflow -> GCP IOs and this is >> expected since Dataflow depends on parts of those implementations for its >> own execution purposes. We definitely don't want GCP IOs depending on >> Dataflow since we would like users of other runners to still be able to use >> GCP IOs without bringing in Dataflow specific dependencies. >> >> There is already a test definition inside of the Dataflow runner package >> that is meant to run integration tests defined in the GCP IO package named >> googleCloudPlatformLegacyWorkerIntegrationTest[1] task, does this fit your >> needs? >> >> 1: >> https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/build.gradle#L318 >> >> On Fri, Jul 12, 2019 at 5:17 AM Michał Walenia < >> michal.wale...@polidea.com> wrote: >> >>> Hi all, >>> recently when I was trying to implement a performance test of >>> BigQueryIO, I ran into an issue when trying to run the test on Dataflow. >>> The problem was that I encountered a circular dependency when compiling >>> the tests. I added the test in org.apache.beam.sdk.io.gcp.bigquery package, >>> so I also needed to add DataflowRunner as a dependency in order to launch >>> the test. The error was that DataflowRunner package depends on >>> org.apache.beam.sdk.io.gcp.bigquery package (for example in [1]). >>> Should it be like that? >>> For now, in order to solve the problem, I intend to move the performance >>> test to its own package in my PR [2] I am wondering about the right >>> approach to this - shouldn’t we decouple the DataflowRunner code from IOs? >>> If not, what’s the reason behind the way the modules are organized? >>> I noticed that 5 tests are excluded from the integrationTest task in >>> io.google-cloud-platform.bigquery build.gradle file [3]. Are they >>> launched on Dataflow anywhere? I couldn’t find their usage except for the >>> exclusions. >>> >>> [1] PubSubIO translations section in DataflowRunner.java >>> <https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L1104> >>> [2] My PR <https://github.com/apache/beam/pull/9041> >>> [3] DefaultCoderCloudObjectTranslatorRegistrar >>> <https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java#L45> >>> >>> Best regards >>> Michal >>> >>> -- >>> >>> Michał Walenia >>> Polidea <https://www.polidea.com/> | Software Engineer >>> >>> M: +48 791 432 002 <+48791432002> >>> E: michal.wale...@polidea.com >>> >>> Unique Tech >>> Check out our projects! <https://www.polidea.com/our-work> >>> >> > > -- > > Michał Walenia > Polidea <https://www.polidea.com/> | Software Engineer > > M: +48 791 432 002 <+48791432002> > E: michal.wale...@polidea.com > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> >
