Re: [BEAM-4814] Add client configuration to aws options #5983

[Ismaël Mejía]

Hello,

Since I have been changing options related parts of S3 I can take a
look, so I will take the R: I will also probably ping Jacob Marble
(the author of S3FileSystem for additional comments), thanks for
bringing this contribution.


On Wed, Jul 18, 2018 at 10:43 PM John Rudolf Lewis  wrote:
>
> I just submitted a PR, my first for this project: 
> https://github.com/apache/beam/pull/5983
>
> It enables one to use the S3FileSystem from behind a filewall where you need 
> an outbound proxy configured.
>
> I want to be able to TextIO.write().to("s3://mybucket/myfile.txt") but I 
> can't unless I configure a proxy.
>
> With this PR you can specify the proxy configuration either via command line:
>
> --clientConfiguration{"proxyHost":"hostname","proxyPort":1234,"proxyUsername":"username","proxyPassword":"password"}
>
> Or in code:
>
> PipelineOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation().create();
> ClientConfiguration clientConfiguration = new ClientConfiguration();
> clientConfiguration.setProxyHost("hostname");
> clientConfiguration.setProxyPort(1234);
> clientConfiguration.setProxyUsername("username");
> clientConfiguration.setProxyPassword("password");
> options.as(AwsOptions.class).setClientConfiguration(clientConfiguration);
>
> The PR auto selected jbonofre, lukecwik, and charmikaramj as reviewers when i 
> created it. The contribution guide suggested that I ask here to see who else 
> I should add as a reviewer.
>
> Please let me know who I should add as reviewers, or any other changes I 
> should make.
>
> Thanks!!

[BEAM-4814] Add client configuration to aws options #5983

[John Rudolf Lewis]

I just submitted a PR, my first for this project:
https://github.com/apache/beam/pull/5983

It enables one to use the S3FileSystem from behind a filewall where you
need an outbound proxy configured.

I want to be able to TextIO.write().to("s3://mybucket/myfile.txt") but I
can't unless I configure a proxy.

With this PR you can specify the proxy configuration either via command
line:

--clientConfiguration{"proxyHost":"hostname","proxyPort":1234,"proxyUsername":"username","proxyPassword":"password"}

Or in code:

PipelineOptions options =
PipelineOptionsFactory.fromArgs(args).withValidation().create();
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setProxyHost("hostname");
clientConfiguration.setProxyPort(1234);
clientConfiguration.setProxyUsername("username");
clientConfiguration.setProxyPassword("password");
options.as(AwsOptions.class).setClientConfiguration(clientConfiguration);

The PR auto selected jbonofre, lukecwik, and charmikaramj as reviewers when
i created it. The contribution guide suggested that I ask here to see who
else I should add as a reviewer.

Please let me know who I should add as reviewers, or any other changes I
should make.

Thanks!!