It would also be ideal if someone extended the SqsIO transform to allow
specifying endpoints via a builder method.
On Fri, Sep 25, 2020 at 9:50 AM Luke Cwik <lc...@google.com> wrote:
> You can create a PipelineOptions interface with a unique endpoint
> identifier for each service you want to use like:
> public interface AwsEndpointOptions extends PipelineOptions {
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getSnsEndpoint();
> void setSnsEndpoint();
>
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getS3Endpoint();
> void setS3Endpoint();
> ...
>
> class FallbackToAwsOptions implements DefaultValueFactory<String> {
> @Override
> public String create(PipelineOptions options) {
> return options.as(AwsOptions.class).getAwsServiceEndpoint();
> }
> }
> }
>
> It looks like AWS IOs support passing in a provider[1] and/or configuring
> the endpoint during pipeline construction. Then during pipeline
> creation/execution you can configure each AWS IO instance with the specific
> endpoint from this new PipelineOptions interface.
>
> 1:
> https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
>
> On Thu, Sep 24, 2020 at 4:24 PM <tclem...@tutanota.com> wrote:
>
>> We're currently working on getting our Beam app working with localstack
>> (and potentially other AWS regions). We're using SqsIO and S3 as part of
>> our pipeline (with other AWS components likely to come into the mix).
>> While I could cast the PipelineOptions to AwsOptions and then call
>> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
>> won't work as different AWS services make use of different endpoints --
>> e.g. the endpoint for SqsIO isn't going to work for S3.
>>
>> What I'd really like to do is provide a different set of AwsOptions per
>> AWS service. What's the best means of accomplishing this?
>>
>> Tim.
>>
>>
