Re: disable ssl check/set verify cert false for elasticsearchIO
It doesn't look like it based upon the javadoc: https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html You can allow self signed certs which might be enough for you though: https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html#withTrustSelfSignedCerts-boolean- Feel free to create a PR and add any options following the pattern that is already there in the code. On Thu, Sep 24, 2020 at 8:36 PM Mohil Khare wrote: > Hello team and elasticSearchIO users, > > I am using beam java sdk 2.23.0 with elastic search as one of the sinks. > > Is there any way to turn off ssl check i.e. set cert verify false for > https connection with elasticsearchIO ? I know using regular clients, you > can do that. But can we achieve the same using elasticsearchIO ? > > Thanks and regards > Mohil >
Re: disable ssl check/set verify cert false for elasticsearchIO
Hi Luke, Yeah, I looked at withTrustSelfSignedCerts options, but after looking at elasticsearchIO code, it seems that flag comes into effect only when "withKeystorePath" is provided. I don't think that can be any gcs path. I believe that is a path in worker VM and for that I will have to use JvmInitliazer to add a custom keystore. Regarding: "Feel free to create a PR and add any options following the pattern that is already there in the code" . What are the guidelines for adding code and creating PR for beam codebase. Can you point me to the relevant document? Thanks and regards Mohil On Fri, Sep 25, 2020 at 8:45 AM Luke Cwik wrote: > It doesn't look like it based upon the javadoc: > > https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html > > You can allow self signed certs which might be enough for you though: > > https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html#withTrustSelfSignedCerts-boolean- > > Feel free to create a PR and add any options following the pattern that is > already there in the code. > > On Thu, Sep 24, 2020 at 8:36 PM Mohil Khare wrote: > >> Hello team and elasticSearchIO users, >> >> I am using beam java sdk 2.23.0 with elastic search as one of the sinks. >> >> Is there any way to turn off ssl check i.e. set cert verify false for >> https connection with elasticsearchIO ? I know using regular clients, you >> can do that. But can we achieve the same using elasticsearchIO ? >> >> Thanks and regards >> Mohil >> >
Re: disable ssl check/set verify cert false for elasticsearchIO
Yeah, the JvmInitializer would be necessary if you wanted to use a file based keystore but I have seen people create in memory keystores[1]. Take a look at the contribution guide[2]. 1: https://gist.github.com/mikeapr4/3b2b5d05bc57640e77d0 2: https://beam.apache.org/contribute/ On Fri, Sep 25, 2020 at 9:17 AM Mohil Khare wrote: > Hi Luke, > Yeah, I looked at withTrustSelfSignedCerts options, but after looking at > elasticsearchIO code, it seems that flag comes into effect only when > "withKeystorePath" is provided. > I don't think that can be any gcs path. I believe that is a path in worker > VM and for that I will have to use JvmInitliazer to add a custom keystore. > > Regarding: "Feel free to create a PR and add any options following the > pattern that is already there in the code" . What are the guidelines for > adding code and creating PR for beam codebase. Can you point me to the > relevant document? > > Thanks and regards > Mohil > > > On Fri, Sep 25, 2020 at 8:45 AM Luke Cwik wrote: > >> It doesn't look like it based upon the javadoc: >> >> https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html >> >> You can allow self signed certs which might be enough for you though: >> >> https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html#withTrustSelfSignedCerts-boolean- >> >> Feel free to create a PR and add any options following the pattern that >> is already there in the code. >> >> On Thu, Sep 24, 2020 at 8:36 PM Mohil Khare wrote: >> >>> Hello team and elasticSearchIO users, >>> >>> I am using beam java sdk 2.23.0 with elastic search as one of the sinks. >>> >>> Is there any way to turn off ssl check i.e. set cert verify false for >>> https connection with elasticsearchIO ? I know using regular clients, you >>> can do that. But can we achieve the same using elasticsearchIO ? >>> >>> Thanks and regards >>> Mohil >>> >>
Re: disable ssl check/set verify cert false for elasticsearchIO
Great.. Thanks a lot for the pointers Luke. Regards Mohil On Fri, Sep 25, 2020 at 9:23 AM Luke Cwik wrote: > Yeah, the JvmInitializer would be necessary if you wanted to use a file > based keystore but I have seen people create in memory keystores[1]. > > Take a look at the contribution guide[2]. > > 1: https://gist.github.com/mikeapr4/3b2b5d05bc57640e77d0 > 2: https://beam.apache.org/contribute/ > > On Fri, Sep 25, 2020 at 9:17 AM Mohil Khare wrote: > >> Hi Luke, >> Yeah, I looked at withTrustSelfSignedCerts options, but after looking at >> elasticsearchIO code, it seems that flag comes into effect only when >> "withKeystorePath" is provided. >> I don't think that can be any gcs path. I believe that is a path in >> worker VM and for that I will have to use JvmInitliazer to add a custom >> keystore. >> >> Regarding: "Feel free to create a PR and add any options following the >> pattern that is already there in the code" . What are the guidelines for >> adding code and creating PR for beam codebase. Can you point me to the >> relevant document? >> >> Thanks and regards >> Mohil >> >> >> On Fri, Sep 25, 2020 at 8:45 AM Luke Cwik wrote: >> >>> It doesn't look like it based upon the javadoc: >>> >>> https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html >>> >>> You can allow self signed certs which might be enough for you though: >>> >>> https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.ConnectionConfiguration.html#withTrustSelfSignedCerts-boolean- >>> >>> Feel free to create a PR and add any options following the pattern that >>> is already there in the code. >>> >>> On Thu, Sep 24, 2020 at 8:36 PM Mohil Khare wrote: >>> Hello team and elasticSearchIO users, I am using beam java sdk 2.23.0 with elastic search as one of the sinks. Is there any way to turn off ssl check i.e. set cert verify false for https connection with elasticsearchIO ? I know using regular clients, you can do that. But can we achieve the same using elasticsearchIO ? Thanks and regards Mohil >>>
Re: Setting AWS endponts per transform
You can create a PipelineOptions interface with a unique endpoint
identifier for each service you want to use like:
public interface AwsEndpointOptions extends PipelineOptions {
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getSnsEndpoint();
void setSnsEndpoint();
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getS3Endpoint();
void setS3Endpoint();
...
class FallbackToAwsOptions implements DefaultValueFactory {
@Override
public String create(PipelineOptions options) {
return options.as(AwsOptions.class).getAwsServiceEndpoint();
}
}
}
It looks like AWS IOs support passing in a provider[1] and/or configuring
the endpoint during pipeline construction. Then during pipeline
creation/execution you can configure each AWS IO instance with the specific
endpoint from this new PipelineOptions interface.
1:
https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
On Thu, Sep 24, 2020 at 4:24 PM wrote:
> We're currently working on getting our Beam app working with localstack
> (and potentially other AWS regions). We're using SqsIO and S3 as part of
> our pipeline (with other AWS components likely to come into the mix).
> While I could cast the PipelineOptions to AwsOptions and then call
> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
> won't work as different AWS services make use of different endpoints --
> e.g. the endpoint for SqsIO isn't going to work for S3.
>
> What I'd really like to do is provide a different set of AwsOptions per
> AWS service. What's the best means of accomplishing this?
>
> Tim.
>
>
Re: Setting AWS endponts per transform
It would also be ideal if someone extended the SqsIO transform to allow
specifying endpoints via a builder method.
On Fri, Sep 25, 2020 at 9:50 AM Luke Cwik wrote:
> You can create a PipelineOptions interface with a unique endpoint
> identifier for each service you want to use like:
> public interface AwsEndpointOptions extends PipelineOptions {
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getSnsEndpoint();
> void setSnsEndpoint();
>
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getS3Endpoint();
> void setS3Endpoint();
> ...
>
> class FallbackToAwsOptions implements DefaultValueFactory {
> @Override
> public String create(PipelineOptions options) {
> return options.as(AwsOptions.class).getAwsServiceEndpoint();
> }
> }
> }
>
> It looks like AWS IOs support passing in a provider[1] and/or configuring
> the endpoint during pipeline construction. Then during pipeline
> creation/execution you can configure each AWS IO instance with the specific
> endpoint from this new PipelineOptions interface.
>
> 1:
> https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
>
> On Thu, Sep 24, 2020 at 4:24 PM wrote:
>
>> We're currently working on getting our Beam app working with localstack
>> (and potentially other AWS regions). We're using SqsIO and S3 as part of
>> our pipeline (with other AWS components likely to come into the mix).
>> While I could cast the PipelineOptions to AwsOptions and then call
>> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
>> won't work as different AWS services make use of different endpoints --
>> e.g. the endpoint for SqsIO isn't going to work for S3.
>>
>> What I'd really like to do is provide a different set of AwsOptions per
>> AWS service. What's the best means of accomplishing this?
>>
>> Tim.
>>
>>
