Thanks for the information. I don't think that the task you mentioned fits my case - I want to run a performance test on a real service and running it with a battery of other tests doesn't make much sense to me. It's similar to the case of other integration tests that are excluded from the task you linked. Is keeping the test in a separate package viable in your opinion?
Thanks! Michal On Fri, Jul 12, 2019 at 3:45 PM Lukasz Cwik <lc...@google.com> wrote: > Yes, there is a dependency between Dataflow -> GCP IOs and this is > expected since Dataflow depends on parts of those implementations for its > own execution purposes. We definitely don't want GCP IOs depending on > Dataflow since we would like users of other runners to still be able to use > GCP IOs without bringing in Dataflow specific dependencies. > > There is already a test definition inside of the Dataflow runner package > that is meant to run integration tests defined in the GCP IO package named > googleCloudPlatformLegacyWorkerIntegrationTest[1] task, does this fit your > needs? > > 1: > https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/build.gradle#L318 > > On Fri, Jul 12, 2019 at 5:17 AM Michał Walenia <michal.wale...@polidea.com> > wrote: > >> Hi all, >> recently when I was trying to implement a performance test of BigQueryIO, >> I ran into an issue when trying to run the test on Dataflow. >> The problem was that I encountered a circular dependency when compiling >> the tests. I added the test in org.apache.beam.sdk.io.gcp.bigquery package, >> so I also needed to add DataflowRunner as a dependency in order to launch >> the test. The error was that DataflowRunner package depends on >> org.apache.beam.sdk.io.gcp.bigquery package (for example in [1]). Should >> it be like that? >> For now, in order to solve the problem, I intend to move the performance >> test to its own package in my PR [2] I am wondering about the right >> approach to this - shouldn’t we decouple the DataflowRunner code from IOs? >> If not, what’s the reason behind the way the modules are organized? >> I noticed that 5 tests are excluded from the integrationTest task in >> io.google-cloud-platform.bigquery build.gradle file [3]. Are they >> launched on Dataflow anywhere? I couldn’t find their usage except for the >> exclusions. >> >> [1] PubSubIO translations section in DataflowRunner.java >> <https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java#L1104> >> [2] My PR <https://github.com/apache/beam/pull/9041> >> [3] DefaultCoderCloudObjectTranslatorRegistrar >> <https://github.com/apache/beam/blob/0fce2b88660f52dae638697e1472aa108c982ae6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java#L45> >> >> Best regards >> Michal >> >> -- >> >> Michał Walenia >> Polidea <https://www.polidea.com/> | Software Engineer >> >> M: +48 791 432 002 <+48791432002> >> E: michal.wale...@polidea.com >> >> Unique Tech >> Check out our projects! <https://www.polidea.com/our-work> >> > -- Michał Walenia Polidea <https://www.polidea.com/> | Software Engineer M: +48 791 432 002 <+48791432002> E: michal.wale...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>
