You can use Create in streaming pipelines as well but you want to ensure
that --streaming is passed as a flag.
You could update the existing test target and force --streaming to be
inserted for example here:
https://github.com/lukecwik/incubator-beam/blob/8097972b4d0ed759aa45f6710ac02b982c6e8deb/runners/google-cloud-dataflow-java/build.gradle#L156
Create can't exercise a lot of complex windowing/triggering semantics
though or can not produce deterministic output which is why TestStream was
created which is like Create but allows you to control watermark and
processing time advancement. This allows for more complicated triggering
and execution but it only works with portable Dataflow pipelines that also
use this additional experiment ('use_unified_worker') which can be launched
with this target:
https://github.com/lukecwik/incubator-beam/blob/8097972b4d0ed759aa45f6710ac02b982c6e8deb/runners/google-cloud-dataflow-java/build.gradle#L246
We are looking to have the first API stable version using the portability
framework with 2.21.0 which should mean that tests that run outside of
Google will be possible.
On Thu, Apr 9, 2020 at 7:17 AM Steve Niemitz <sniem...@apache.org> wrote:
> I was trying to run a @ValidatesRunner test for the streaming dataflow
> runner, but I actually can't find any way to run them in streaming. It
> looks like all the tests are set up using the Create transform,
> which generates a batch pipeline.
>
> Are there actually no @ValidatesRunner tests for the streaming dataflow
> runner? That seems like a big gap in coverage. Am I missing a setting
> somewhere?
>
