+dev@beam.apache.org <dev@beam.apache.org>
Digging more into the code it looks more like a bug that was introduced in
a recent PR (https://github.com/apache/beam/pull/13180). It seems that
pipeline_options is absent when using external workers and it causes the
S3IO to fail in an assertion in its constructor.
I wonder what is the reason this assertion was added as it seems that it
hasn't been there in the past.
Any assistance would be appreciated!
Nir
On Wed, Feb 10, 2021 at 6:49 PM Nir Gazit <nir....@gmail.com> wrote:
> Hey,
> I'm getting this error:
> apache_beam.io.filesystem.BeamIOError: Match operation failed with
> exceptions {'s3://fiverr-data-science-de
> v/beam_poc/beam/wc/input.txt': BeamIOError("exists() operation failed with
> exceptions {'s3://fiverr-data-sc
> ience-dev/beam_poc/beam/wc/input.txt': ValueError('Must provide one of
> client or options')}")} [while runni
> ng
> 'ReadFromText/Read/SDFBoundedSourceReader/ParDo(SDFBoundedSourceDoFn)/PairWithRestriction0']
> with except
> ions None
>
> When trying to run a simple pipeline over Flink with an external
> environment.
>
> What could be the problem?
>
> Thanks!
>
