By separate package, I mean a distinct Gradle module with a separate set of dependencies. I know about the movement away from Perfkit and I plan to create Jenkins jobs utilizing only Gradle tasks for the tests. Lukasz, should I add you as a reviewer for the pull request?
Thank you for the resources. Michal On Mon, Jul 15, 2019 at 3:50 PM Lukasz Cwik <lc...@google.com> wrote: > When do you mean by separate package? > > Most of our perf tests are integrated with PerfkitBenchmarker such as our > JDBC benchmark[1] and Python BigQuery benchmark[2] and point to a specific > IT class/method that contains the benchmarks. > > Note that there was a recent discussion about moving away from > PerfkitBenchmarker though and it would be good to follow up with Lukasz > Gajowy about it since he has done a bunch of the existing benchmarks[3]. > > Finally, there were some docs about benchmarking on our design docs[4] > page that may provide additional details[5, 6]. > > 1: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_JDBC.groovy > 2: > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy > 3: > https://lists.apache.org/thread.html/dab1c093799248787e8b75e63b66d7389b594b649a4d9a4a5db1cfbb@%3Cdev.beam.apache.org%3E > 4: https://beam.apache.org/contribute/design-documents/ > 5: > https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE > 6: > https://docs.google.com/document/d/1Cb7XVmqe__nA_WCrriAifL-3WCzbZzV4Am5W_SkQLeA > > On Mon, Jul 15, 2019 at 8:03 AM Michał Walenia <michal.wale...@polidea.com> > wrote: > >> Thanks for the information. I don't think that the task you mentioned >> fits my case - I want to run a performance test on a real service and >> running it with a battery of other tests doesn't make much sense to me. >> It's similar to the case of other integration tests that are excluded >> from the task you linked. >> Is keeping the test in a separate package viable in your opinion? >> >> Thanks! >> Michal >> >> On Fri, Jul 12, 2019 at 3:45 PM Lukasz Cwik <lc...@google.com> wrote: >> >>> Yes, there is a dependency between Dataflow -> GCP IOs and this is >>> expected since Dataflow depends on parts of those implementations for its >>> own execution purposes. We definitely don't want GCP IOs depending on >>> Dataflow since we would like users of other runners to still be able to use >>> GCP IOs without bringing in Dataflow specific dependencies. >>> >>> There is already a test definition inside of the Dataflow runner package >>> that is meant to run integration tests defined in the GCP IO package named >>> googleCloudPlatformLegacyWorkerIntegrationTest[1] task, does this fit your >>> needs? >>> >>> 1: >>> https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/build.gradle#L318 >>> >>> On Fri, Jul 12, 2019 at 5:17 AM Michał Walenia < >>> michal.wale...@polidea.com> wrote: >>> >>>> Hi all, >>>> recently when I was trying to implement a performance test of >>>> BigQueryIO, I ran into an issue when trying to run the test on Dataflow. >>>> The problem was that I encountered a circular dependency when compiling >>>> the tests. I added the test in org.apache.beam.sdk.io.gcp.bigquery package, >>>> so I also needed to add DataflowRunner as a dependency in order to launch >>>> the test. The error was that DataflowRunner package depends on >>>> org.apache.beam.sdk.io.gcp.bigquery package (for example in [1]). >>>> Should it be like that? >>>> For now, in order to solve the problem, I intend to move the >>>> performance test to its own package in my PR [2] I am wondering about the >>>> right approach to this - shouldn’t we decouple the DataflowRunner code from >>>> IOs? If not, what’s the reason behind the way the modules are organized? >>>> I noticed that 5 tests are excluded from the integrationTest task in >>>> io.google-cloud-platform.bigquery build.gradle file [3]. Are they >>>> launched on Dataflow anywhere? I couldn’t find their usage except for the >>>> exclusions. >>>> >>>> [1] PubSubIO translations section in DataflowRunner.java >>>> <https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L1104> >>>> [2] My PR <https://github.com/apache/beam/pull/9041> >>>> [3] DefaultCoderCloudObjectTranslatorRegistrar >>>> <https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java#L45> >>>> >>>> Best regards >>>> Michal >>>> >>>> -- >>>> >>>> Michał Walenia >>>> Polidea <https://www.polidea.com/> | Software Engineer >>>> >>>> M: +48 791 432 002 <+48791432002> >>>> E: michal.wale...@polidea.com >>>> >>>> Unique Tech >>>> Check out our projects! <https://www.polidea.com/our-work> >>>> >>> >> >> -- >> >> Michał Walenia >> Polidea <https://www.polidea.com/> | Software Engineer >> >> M: +48 791 432 002 <+48791432002> >> E: michal.wale...@polidea.com >> >> Unique Tech >> Check out our projects! <https://www.polidea.com/our-work> >> > -- Michał Walenia Polidea <https://www.polidea.com/> | Software Engineer M: +48 791 432 002 <+48791432002> E: michal.wale...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
