Hi all,
Looking for help to understand an internal error from beam dataflow runner.
I have a streaming pipeline that is on google dataflow runner(beam version
2.11, java). Recently I added a SplittableDoFn to my pipeline to
continuously generate a sequence. However, after the job runs for a few
h
This is a known issue and was fixed with
https://github.com/apache/beam/commit/5d9bb4595c763025a369a959e18c6dd288e72314#diff-f149847d2c06f56ea591cab8d862c960
It is meant to be released as part of 2.16.0
On Tue, Aug 27, 2019 at 11:41 AM Zhiheng Huang
wrote:
> Hi all,
>
> Looking for help to unde
Thanks! Glad that this will be fixed in future release.
Is there anyway that I can avoid hitting this problem before 2.16 is
released?
On Tue, Aug 27, 2019 at 12:57 PM Lukasz Cwik wrote:
> This is a known issue and was fixed with
> https://github.com/apache/beam/commit/5d9bb4595c763025a369a959e
You could clone the 2.X.0 branch of the Beam repo and patch in the commit I
referenced above. Then build the gradle target
:runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar
which should produce a jar in the build/libs/ directory. Pass the
additional flag --dataflowWorkerJar=path/to
Thanks!
On Tue, Aug 27, 2019 at 2:14 PM Lukasz Cwik wrote:
> You could clone the 2.X.0 branch of the Beam repo and patch in the commit
> I referenced above. Then build the gradle target
> :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar
> which should produce a jar in the build
