[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687677#comment-16687677
]
ASF GitHub Bot commented on FLINK-10628:
zentol closed pull request #6967: [FLINK-10628][E2E][SSL] Enable mutual REST
SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/ops/security-ssl.md b/docs/ops/security-ssl.md
index 4e3716218d2..77099ebf61f 100644
--- a/docs/ops/security-ssl.md
+++ b/docs/ops/security-ssl.md
@@ -198,7 +198,7 @@ This example shows how to create a simple keystore /
truststore pair. The trusts
be shared with other applications. In this example, *myhost.company.org /
ip:10.0.2.15* is the node (or service) for the Flink master.
{% highlight bash %}
-keytool -genkeypair -alias flink.rest -keystore rest.keystore -dname
"CN=myhost.company.org" -ext "SAN=dns:myhost.company.org,ip:10.0.2.15"
-storepass rest_keystore_password -keypass rest_key_password -keyalg RSA
-keysize 4096
+keytool -genkeypair -alias flink.rest -keystore rest.keystore -dname
"CN=myhost.company.org" -ext "SAN=dns:myhost.company.org,ip:10.0.2.15"
-storepass rest_keystore_password -keypass rest_key_password -keyalg RSA
-keysize 4096 -storetype PKCS12
keytool -exportcert -keystore rest.keystore -alias flink.rest -storepass
rest_keystore_password -file flink.cer
@@ -219,7 +219,7 @@ security.ssl.rest.key-password: rest_key_password
Execute the following keytool commands to create a truststore with a self
signed CA.
{% highlight bash %}
-keytool -genkeypair -alias ca -keystore ca.keystore -dname "CN=Sample CA"
-storepass ca_keystore_password -keypass ca_key_password -keyalg RSA -keysize
4096 -ext "bc=ca:true"
+keytool -genkeypair -alias ca -keystore ca.keystore -dname "CN=Sample CA"
-storepass ca_keystore_password -keypass ca_key_password -keyalg RSA -keysize
4096 -ext "bc=ca:true" -storetype PKCS12
keytool -exportcert -keystore ca.keystore -alias ca -storepass
ca_keystore_password -file ca.cer
@@ -230,7 +230,7 @@ Now create a keystore for the REST endpoint with a
certificate signed by the abo
Let *flink.company.org / ip:10.0.2.15* be the hostname of the Flink master
(JobManager).
{% highlight bash %}
-keytool -genkeypair -alias flink.rest -keystore rest.signed.keystore -dname
"CN=flink.company.org" -ext "SAN=dns:flink.company.org" -storepass
rest_keystore_password -keypass rest_key_password -keyalg RSA -keysize 4096
+keytool -genkeypair -alias flink.rest -keystore rest.signed.keystore -dname
"CN=flink.company.org" -ext "SAN=dns:flink.company.org" -storepass
rest_keystore_password -keypass rest_key_password -keyalg RSA -keysize 4096
-storetype PKCS12
keytool -certreq -alias flink.rest -keystore rest.signed.keystore -storepass
rest_keystore_password -keypass rest_key_password -file rest.csr
@@ -252,6 +252,25 @@ security.ssl.rest.key-password: rest_key_password
security.ssl.rest.truststore-password: ca_truststore_password
{% endhighlight %}
+**Tips to query REST Endpoint with curl utility**
+
+You can convert the keystore into the `PEM` format using `openssl`:
+
+{% highlight bash %}
+openssl pkcs12 -passin pass:rest_keystore_password -in rest.keystore -out
rest.pem -nodes
+{% endhighlight %}
+
+Then you can query REST Endpoint with `curl`:
+
+{% highlight bash %}
+curl --cacert rest.pem flink_url
+{% endhighlight %}
+
+If mutual SSL is enabled:
+
+{% highlight bash %}
+curl --cacert rest.pem --cert rest.pem flink_url
+{% endhighlight %}
## Tips for YARN / Mesos Deployment
diff --git a/flink-end-to-end-tests/test-scripts/common.sh
b/flink-end-to-end-tests/test-scripts/common.sh
index 0024d82f3d6..ea267538639 100644
--- a/flink-end-to-end-tests/test-scripts/common.sh
+++ b/flink-end-to-end-tests/test-scripts/common.sh
@@ -41,13 +41,20 @@ echo "Flink dist directory: $FLINK_DIR"
FLINK_VERSION=$(cat ${END_TO_END_DIR}/pom.xml | sed -n
's/.*\(.*\)<\/version>/\1/p')
-USE_SSL=OFF # set via set_conf_ssl(), reset via revert_default_config()
TEST_ROOT=`pwd -P`
TEST_INFRA_DIR="$END_TO_END_DIR/test-scripts/"
cd $TEST_INFRA_DIR
TEST_INFRA_DIR=`pwd -P`
cd $TEST_ROOT
+NODENAME=`hostname -f`
+
+# REST_PROTOCOL and CURL_SSL_ARGS can be modified in common_ssl.sh if SSL is
activated
+# they should be used in curl command to query Flink REST API
+REST_PROTOCOL="http"
+CURL_SSL_ARGS=""
+source "${TEST_INFRA_DIR}/common_ssl.sh"
+
function print_mem_use_osx {
declare -a mem_types=("active" "inactive" "wired down")
used=""
@@ -87,7 +94,8 @@ function revert_default_config() {
mv -f $FLINK_DIR/conf/flink-conf.yaml.bak
$FLINK_DIR/conf/flink-conf.yaml
fi
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[ https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686319#comment-16686319 ] ASF GitHub Bot commented on FLINK-10628: zentol commented on issue #6967: [FLINK-10628][E2E][SSL] Enable mutual REST SSL auth in e2e tests URL: https://github.com/apache/flink/pull/6967#issuecomment-438613591 Please rebase the branch, e2e test runs are still failing but this is likely just caused by the recent instabilities in general. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add end-to-end test for REST communication encryption > - > > Key: FLINK-10628 > URL: https://issues.apache.org/jira/browse/FLINK-10628 > Project: Flink > Issue Type: Sub-task > Components: REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > With FLINK-10371 we added support for mutual authentication for the REST > communication. We should adapt one of the existing end-to-end tests to > require this feature for the REST communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[ https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681477#comment-16681477 ] ASF GitHub Bot commented on FLINK-10628: azagrebin commented on issue #6967: [FLINK-10628][E2E][SSL] Enable mutual REST SSL auth in e2e tests URL: https://github.com/apache/flink/pull/6967#issuecomment-437369228 I reverted the loop changes and factored out them into another PR #7073 for new issue FLINK-10842. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add end-to-end test for REST communication encryption > - > > Key: FLINK-10628 > URL: https://issues.apache.org/jira/browse/FLINK-10628 > Project: Flink > Issue Type: Sub-task > Components: REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > With FLINK-10371 we added support for mutual authentication for the REST > communication. We should adapt one of the existing end-to-end tests to > require this feature for the REST communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[ https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681479#comment-16681479 ] ASF GitHub Bot commented on FLINK-10628: azagrebin edited a comment on issue #6967: [FLINK-10628][E2E][SSL] Enable mutual REST SSL auth in e2e tests URL: https://github.com/apache/flink/pull/6967#issuecomment-437369228 I reverted the loop changes and factored out them into another PR #7073 for new issue [FLINK-10842](https://issues.apache.org/jira/browse/FLINK-10842). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add end-to-end test for REST communication encryption > - > > Key: FLINK-10628 > URL: https://issues.apache.org/jira/browse/FLINK-10628 > Project: Flink > Issue Type: Sub-task > Components: REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > With FLINK-10371 we added support for mutual authentication for the REST > communication. We should adapt one of the existing end-to-end tests to > require this feature for the REST communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[ https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681353#comment-16681353 ] ASF GitHub Bot commented on FLINK-10628: zentol commented on issue #6967: [FLINK-10628][E2E][SSL] Enable mutual REST SSL auth in e2e tests URL: https://github.com/apache/flink/pull/6967#issuecomment-437339953 Update: The loop changes are causing some test failures. Please open a separate JIRA as this is probably a larger issue. Can we cut the loop changes out of this PR without breaking the test? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add end-to-end test for REST communication encryption > - > > Key: FLINK-10628 > URL: https://issues.apache.org/jira/browse/FLINK-10628 > Project: Flink > Issue Type: Sub-task > Components: REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > With FLINK-10371 we added support for mutual authentication for the REST > communication. We should adapt one of the existing end-to-end tests to > require this feature for the REST communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671431#comment-16671431
]
ASF GitHub Bot commented on FLINK-10628:
zentol commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229997416
##
File path: docs/ops/security-ssl.md
##
@@ -252,6 +252,25 @@ security.ssl.rest.key-password: rest_key_password
security.ssl.rest.truststore-password: ca_truststore_password
{% endhighlight %}
+**Tips to query REST Endpoint with curl utility**
+
+You can generate convert the keystore into the `PEM` format using `openssl`:
+
+{% highlight bash %}
+openssl pkcs12 -passin pass:rest_keystore_password -in rest.keystore -out
rest.pem -nodes
+{% endhighlight %}
+
+Then you can query REST Endpoint with `curl`:
+
+{% highlight bash %}
+curl --cacert rest.pem flink_url
+{% endhighlight %}
+
+If mutual SSL is enabled:
+
+{% highlight bash %}
+curl --cacert rest.pem --cert rest.pem flink_url
Review comment:
remove second space after `--cacert rest.pem`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add end-to-end test for REST communication encryption
> -
>
> Key: FLINK-10628
> URL: https://issues.apache.org/jira/browse/FLINK-10628
> Project: Flink
> Issue Type: Sub-task
> Components: REST
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> With FLINK-10371 we added support for mutual authentication for the REST
> communication. We should adapt one of the existing end-to-end tests to
> require this feature for the REST communication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671432#comment-16671432
]
ASF GitHub Bot commented on FLINK-10628:
zentol commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229997184
##
File path: docs/ops/security-ssl.md
##
@@ -252,6 +252,25 @@ security.ssl.rest.key-password: rest_key_password
security.ssl.rest.truststore-password: ca_truststore_password
{% endhighlight %}
+**Tips to query REST Endpoint with curl utility**
+
+You can generate convert the keystore into the `PEM` format using `openssl`:
Review comment:
remove `generate`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add end-to-end test for REST communication encryption
> -
>
> Key: FLINK-10628
> URL: https://issues.apache.org/jira/browse/FLINK-10628
> Project: Flink
> Issue Type: Sub-task
> Components: REST
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> With FLINK-10371 we added support for mutual authentication for the REST
> communication. We should adapt one of the existing end-to-end tests to
> require this feature for the REST communication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671414#comment-16671414
]
ASF GitHub Bot commented on FLINK-10628:
azagrebin commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229995637
##
File path: flink-end-to-end-tests/test-scripts/common_ssl.sh
##
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+function _set_conf_ssl_helper {
+local type=$1 # 'internal' or external 'rest'
+local ssl_dir="${TEST_DATA_DIR}/ssl/${type}"
+local password="${type}.password"
+
+if [ "${type}" != "internal" ] && [ "${type}" != "rest" ]; then
+echo "Unknown type of ssl connectivity: ${type}. It can be either
'internal' or external 'rest'"
+exit 1
+fi
+
+# clean up the dir that will be used for SSL certificates and trust stores
+if [ -e "${ssl_dir}" ]; then
+ echo "File ${ssl_dir} exists. Deleting it..."
+ rm -rf "${ssl_dir}"
+fi
+mkdir -p "${ssl_dir}"
+
+SANSTRING="dns:${NODENAME}"
+for NODEIP in $(get_node_ip) ; do
+SANSTRING="${SANSTRING},ip:${NODEIP}"
+done
+
+echo "Using SAN ${SANSTRING}"
+
+# create certificates
+keytool -genkeypair -alias ca -keystore "${ssl_dir}/ca.keystore" -dname
"CN=Sample CA" -storepass ${password} -keypass ${password} -keyalg RSA -ext
bc=ca:true -storetype PKCS12
+keytool -keystore "${ssl_dir}/ca.keystore" -storepass ${password} -alias
ca -exportcert > "${ssl_dir}/ca.cer"
+keytool -importcert -keystore "${ssl_dir}/ca.truststore" -alias ca
-storepass ${password} -noprompt -file "${ssl_dir}/ca.cer"
+
+keytool -genkeypair -alias node -keystore "${ssl_dir}/node.keystore"
-dname "CN=${NODENAME}" -ext SAN=${SANSTRING} -storepass ${password} -keypass
${password} -keyalg RSA -storetype PKCS12
+keytool -certreq -keystore "${ssl_dir}/node.keystore" -storepass
${password} -alias node -file "${ssl_dir}/node.csr"
+keytool -gencert -keystore "${ssl_dir}/ca.keystore" -storepass ${password}
-alias ca -ext SAN=${SANSTRING} -infile "${ssl_dir}/node.csr" -outfile
"${ssl_dir}/node.cer"
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+
+openssl pkcs12 -passin pass:${password} -in "${ssl_dir}/node.keystore"
-passout pass: -out "${ssl_dir}/node.pem" -nodes
+
+# adapt config
+# (here we rely on security.ssl.enabled enabling SSL for all components
and internal as well as
+# external communication channels)
+set_conf security.ssl.${type}.enabled true
+set_conf security.ssl.${type}.keystore ${ssl_dir}/node.keystore
+set_conf security.ssl.${type}.keystore-password ${password}
+set_conf security.ssl.${type}.key-password ${password}
+set_conf security.ssl.${type}.truststore ${ssl_dir}/ca.truststore
+set_conf security.ssl.${type}.truststore-password ${password}
+}
+
+function set_conf_internal_ssl {
+_set_conf_ssl_helper "internal"
+}
+
+function set_conf_mutual_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local mutual="false"
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+if [ "${auth}" == "mutual" ]; then
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cert ${ssl_dir}/node.pem"
+mutual="true";
+fi
+echo "Mutual ssl auth: ${mutual}"
+set_conf security.ssl.rest.authentication-enabled ${mutual}
+}
+
+function set_conf_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+_set_conf_ssl_helper "rest"
+set_conf_mutual_rest_ssl ${auth}
+REST_PROTOCOL="https"
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cacert
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671411#comment-16671411
]
ASF GitHub Bot commented on FLINK-10628:
azagrebin commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229995229
##
File path: flink-end-to-end-tests/test-scripts/common_ssl.sh
##
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+function _set_conf_ssl_helper {
+local type=$1 # 'internal' or external 'rest'
+local ssl_dir="${TEST_DATA_DIR}/ssl/${type}"
+local password="${type}.password"
+
+if [ "${type}" != "internal" ] && [ "${type}" != "rest" ]; then
+echo "Unknown type of ssl connectivity: ${type}. It can be either
'internal' or external 'rest'"
+exit 1
+fi
+
+# clean up the dir that will be used for SSL certificates and trust stores
+if [ -e "${ssl_dir}" ]; then
+ echo "File ${ssl_dir} exists. Deleting it..."
+ rm -rf "${ssl_dir}"
+fi
+mkdir -p "${ssl_dir}"
+
+SANSTRING="dns:${NODENAME}"
+for NODEIP in $(get_node_ip) ; do
+SANSTRING="${SANSTRING},ip:${NODEIP}"
+done
+
+echo "Using SAN ${SANSTRING}"
+
+# create certificates
+keytool -genkeypair -alias ca -keystore "${ssl_dir}/ca.keystore" -dname
"CN=Sample CA" -storepass ${password} -keypass ${password} -keyalg RSA -ext
bc=ca:true -storetype PKCS12
+keytool -keystore "${ssl_dir}/ca.keystore" -storepass ${password} -alias
ca -exportcert > "${ssl_dir}/ca.cer"
+keytool -importcert -keystore "${ssl_dir}/ca.truststore" -alias ca
-storepass ${password} -noprompt -file "${ssl_dir}/ca.cer"
+
+keytool -genkeypair -alias node -keystore "${ssl_dir}/node.keystore"
-dname "CN=${NODENAME}" -ext SAN=${SANSTRING} -storepass ${password} -keypass
${password} -keyalg RSA -storetype PKCS12
+keytool -certreq -keystore "${ssl_dir}/node.keystore" -storepass
${password} -alias node -file "${ssl_dir}/node.csr"
+keytool -gencert -keystore "${ssl_dir}/ca.keystore" -storepass ${password}
-alias ca -ext SAN=${SANSTRING} -infile "${ssl_dir}/node.csr" -outfile
"${ssl_dir}/node.cer"
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+
+openssl pkcs12 -passin pass:${password} -in "${ssl_dir}/node.keystore"
-passout pass: -out "${ssl_dir}/node.pem" -nodes
+
+# adapt config
+# (here we rely on security.ssl.enabled enabling SSL for all components
and internal as well as
+# external communication channels)
+set_conf security.ssl.${type}.enabled true
+set_conf security.ssl.${type}.keystore ${ssl_dir}/node.keystore
+set_conf security.ssl.${type}.keystore-password ${password}
+set_conf security.ssl.${type}.key-password ${password}
+set_conf security.ssl.${type}.truststore ${ssl_dir}/ca.truststore
+set_conf security.ssl.${type}.truststore-password ${password}
+}
+
+function set_conf_internal_ssl {
+_set_conf_ssl_helper "internal"
+}
+
+function set_conf_mutual_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local mutual="false"
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+if [ "${auth}" == "mutual" ]; then
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cert ${ssl_dir}/node.pem"
+mutual="true";
+fi
+echo "Mutual ssl auth: ${mutual}"
+set_conf security.ssl.rest.authentication-enabled ${mutual}
+}
+
+function set_conf_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+_set_conf_ssl_helper "rest"
+set_conf_mutual_rest_ssl ${auth}
+REST_PROTOCOL="https"
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cacert
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671409#comment-16671409
]
ASF GitHub Bot commented on FLINK-10628:
azagrebin commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229995108
##
File path: flink-end-to-end-tests/test-scripts/common_ssl.sh
##
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+function _set_conf_ssl_helper {
+local type=$1 # 'internal' or external 'rest'
+local ssl_dir="${TEST_DATA_DIR}/ssl/${type}"
+local password="${type}.password"
+
+if [ "${type}" != "internal" ] && [ "${type}" != "rest" ]; then
+echo "Unknown type of ssl connectivity: ${type}. It can be either
'internal' or external 'rest'"
+exit 1
+fi
+
+# clean up the dir that will be used for SSL certificates and trust stores
+if [ -e "${ssl_dir}" ]; then
+ echo "File ${ssl_dir} exists. Deleting it..."
+ rm -rf "${ssl_dir}"
+fi
+mkdir -p "${ssl_dir}"
+
+SANSTRING="dns:${NODENAME}"
+for NODEIP in $(get_node_ip) ; do
+SANSTRING="${SANSTRING},ip:${NODEIP}"
+done
+
+echo "Using SAN ${SANSTRING}"
+
+# create certificates
+keytool -genkeypair -alias ca -keystore "${ssl_dir}/ca.keystore" -dname
"CN=Sample CA" -storepass ${password} -keypass ${password} -keyalg RSA -ext
bc=ca:true -storetype PKCS12
+keytool -keystore "${ssl_dir}/ca.keystore" -storepass ${password} -alias
ca -exportcert > "${ssl_dir}/ca.cer"
+keytool -importcert -keystore "${ssl_dir}/ca.truststore" -alias ca
-storepass ${password} -noprompt -file "${ssl_dir}/ca.cer"
+
+keytool -genkeypair -alias node -keystore "${ssl_dir}/node.keystore"
-dname "CN=${NODENAME}" -ext SAN=${SANSTRING} -storepass ${password} -keypass
${password} -keyalg RSA -storetype PKCS12
+keytool -certreq -keystore "${ssl_dir}/node.keystore" -storepass
${password} -alias node -file "${ssl_dir}/node.csr"
+keytool -gencert -keystore "${ssl_dir}/ca.keystore" -storepass ${password}
-alias ca -ext SAN=${SANSTRING} -infile "${ssl_dir}/node.csr" -outfile
"${ssl_dir}/node.cer"
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+
+openssl pkcs12 -passin pass:${password} -in "${ssl_dir}/node.keystore"
-passout pass: -out "${ssl_dir}/node.pem" -nodes
Review comment:
keystore is converted into the PEM format to use it with curl, I will add a
comment for that
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add end-to-end test for REST communication encryption
> -
>
> Key: FLINK-10628
> URL: https://issues.apache.org/jira/browse/FLINK-10628
> Project: Flink
> Issue Type: Sub-task
> Components: REST
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> With FLINK-10371 we added support for mutual authentication for the REST
> communication. We should adapt one of the existing end-to-end tests to
> require this feature for the REST communication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671277#comment-16671277
]
ASF GitHub Bot commented on FLINK-10628:
zentol commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229972405
##
File path: flink-end-to-end-tests/test-scripts/common_ssl.sh
##
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+function _set_conf_ssl_helper {
+local type=$1 # 'internal' or external 'rest'
+local ssl_dir="${TEST_DATA_DIR}/ssl/${type}"
+local password="${type}.password"
+
+if [ "${type}" != "internal" ] && [ "${type}" != "rest" ]; then
+echo "Unknown type of ssl connectivity: ${type}. It can be either
'internal' or external 'rest'"
+exit 1
+fi
+
+# clean up the dir that will be used for SSL certificates and trust stores
+if [ -e "${ssl_dir}" ]; then
+ echo "File ${ssl_dir} exists. Deleting it..."
+ rm -rf "${ssl_dir}"
+fi
+mkdir -p "${ssl_dir}"
+
+SANSTRING="dns:${NODENAME}"
+for NODEIP in $(get_node_ip) ; do
+SANSTRING="${SANSTRING},ip:${NODEIP}"
+done
+
+echo "Using SAN ${SANSTRING}"
+
+# create certificates
+keytool -genkeypair -alias ca -keystore "${ssl_dir}/ca.keystore" -dname
"CN=Sample CA" -storepass ${password} -keypass ${password} -keyalg RSA -ext
bc=ca:true -storetype PKCS12
+keytool -keystore "${ssl_dir}/ca.keystore" -storepass ${password} -alias
ca -exportcert > "${ssl_dir}/ca.cer"
+keytool -importcert -keystore "${ssl_dir}/ca.truststore" -alias ca
-storepass ${password} -noprompt -file "${ssl_dir}/ca.cer"
+
+keytool -genkeypair -alias node -keystore "${ssl_dir}/node.keystore"
-dname "CN=${NODENAME}" -ext SAN=${SANSTRING} -storepass ${password} -keypass
${password} -keyalg RSA -storetype PKCS12
+keytool -certreq -keystore "${ssl_dir}/node.keystore" -storepass
${password} -alias node -file "${ssl_dir}/node.csr"
+keytool -gencert -keystore "${ssl_dir}/ca.keystore" -storepass ${password}
-alias ca -ext SAN=${SANSTRING} -infile "${ssl_dir}/node.csr" -outfile
"${ssl_dir}/node.cer"
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+
+openssl pkcs12 -passin pass:${password} -in "${ssl_dir}/node.keystore"
-passout pass: -out "${ssl_dir}/node.pem" -nodes
+
+# adapt config
+# (here we rely on security.ssl.enabled enabling SSL for all components
and internal as well as
+# external communication channels)
+set_conf security.ssl.${type}.enabled true
+set_conf security.ssl.${type}.keystore ${ssl_dir}/node.keystore
+set_conf security.ssl.${type}.keystore-password ${password}
+set_conf security.ssl.${type}.key-password ${password}
+set_conf security.ssl.${type}.truststore ${ssl_dir}/ca.truststore
+set_conf security.ssl.${type}.truststore-password ${password}
+}
+
+function set_conf_internal_ssl {
+_set_conf_ssl_helper "internal"
+}
+
+function set_conf_mutual_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local mutual="false"
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+if [ "${auth}" == "mutual" ]; then
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cert ${ssl_dir}/node.pem"
+mutual="true";
+fi
+echo "Mutual ssl auth: ${mutual}"
+set_conf security.ssl.rest.authentication-enabled ${mutual}
+}
+
+function set_conf_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+_set_conf_ssl_helper "rest"
+set_conf_mutual_rest_ssl ${auth}
+REST_PROTOCOL="https"
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cacert
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671278#comment-16671278
]
ASF GitHub Bot commented on FLINK-10628:
zentol commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229971488
##
File path: flink-end-to-end-tests/test-scripts/common_ssl.sh
##
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+function _set_conf_ssl_helper {
+local type=$1 # 'internal' or external 'rest'
+local ssl_dir="${TEST_DATA_DIR}/ssl/${type}"
+local password="${type}.password"
+
+if [ "${type}" != "internal" ] && [ "${type}" != "rest" ]; then
+echo "Unknown type of ssl connectivity: ${type}. It can be either
'internal' or external 'rest'"
+exit 1
+fi
+
+# clean up the dir that will be used for SSL certificates and trust stores
+if [ -e "${ssl_dir}" ]; then
+ echo "File ${ssl_dir} exists. Deleting it..."
+ rm -rf "${ssl_dir}"
+fi
+mkdir -p "${ssl_dir}"
+
+SANSTRING="dns:${NODENAME}"
+for NODEIP in $(get_node_ip) ; do
+SANSTRING="${SANSTRING},ip:${NODEIP}"
+done
+
+echo "Using SAN ${SANSTRING}"
+
+# create certificates
+keytool -genkeypair -alias ca -keystore "${ssl_dir}/ca.keystore" -dname
"CN=Sample CA" -storepass ${password} -keypass ${password} -keyalg RSA -ext
bc=ca:true -storetype PKCS12
+keytool -keystore "${ssl_dir}/ca.keystore" -storepass ${password} -alias
ca -exportcert > "${ssl_dir}/ca.cer"
+keytool -importcert -keystore "${ssl_dir}/ca.truststore" -alias ca
-storepass ${password} -noprompt -file "${ssl_dir}/ca.cer"
+
+keytool -genkeypair -alias node -keystore "${ssl_dir}/node.keystore"
-dname "CN=${NODENAME}" -ext SAN=${SANSTRING} -storepass ${password} -keypass
${password} -keyalg RSA -storetype PKCS12
+keytool -certreq -keystore "${ssl_dir}/node.keystore" -storepass
${password} -alias node -file "${ssl_dir}/node.csr"
+keytool -gencert -keystore "${ssl_dir}/ca.keystore" -storepass ${password}
-alias ca -ext SAN=${SANSTRING} -infile "${ssl_dir}/node.csr" -outfile
"${ssl_dir}/node.cer"
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+
+openssl pkcs12 -passin pass:${password} -in "${ssl_dir}/node.keystore"
-passout pass: -out "${ssl_dir}/node.pem" -nodes
+
+# adapt config
+# (here we rely on security.ssl.enabled enabling SSL for all components
and internal as well as
+# external communication channels)
+set_conf security.ssl.${type}.enabled true
+set_conf security.ssl.${type}.keystore ${ssl_dir}/node.keystore
+set_conf security.ssl.${type}.keystore-password ${password}
+set_conf security.ssl.${type}.key-password ${password}
+set_conf security.ssl.${type}.truststore ${ssl_dir}/ca.truststore
+set_conf security.ssl.${type}.truststore-password ${password}
+}
+
+function set_conf_internal_ssl {
Review comment:
function seems unnecessary
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add end-to-end test for REST communication encryption
> -
>
> Key: FLINK-10628
> URL: https://issues.apache.org/jira/browse/FLINK-10628
> Project: Flink
> Issue Type: Sub-task
> Components: REST
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671276#comment-16671276
]
ASF GitHub Bot commented on FLINK-10628:
zentol commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229971085
##
File path: flink-end-to-end-tests/test-scripts/common_ssl.sh
##
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+function _set_conf_ssl_helper {
+local type=$1 # 'internal' or external 'rest'
+local ssl_dir="${TEST_DATA_DIR}/ssl/${type}"
+local password="${type}.password"
+
+if [ "${type}" != "internal" ] && [ "${type}" != "rest" ]; then
+echo "Unknown type of ssl connectivity: ${type}. It can be either
'internal' or external 'rest'"
+exit 1
+fi
+
+# clean up the dir that will be used for SSL certificates and trust stores
+if [ -e "${ssl_dir}" ]; then
+ echo "File ${ssl_dir} exists. Deleting it..."
+ rm -rf "${ssl_dir}"
+fi
+mkdir -p "${ssl_dir}"
+
+SANSTRING="dns:${NODENAME}"
+for NODEIP in $(get_node_ip) ; do
+SANSTRING="${SANSTRING},ip:${NODEIP}"
+done
+
+echo "Using SAN ${SANSTRING}"
+
+# create certificates
+keytool -genkeypair -alias ca -keystore "${ssl_dir}/ca.keystore" -dname
"CN=Sample CA" -storepass ${password} -keypass ${password} -keyalg RSA -ext
bc=ca:true -storetype PKCS12
+keytool -keystore "${ssl_dir}/ca.keystore" -storepass ${password} -alias
ca -exportcert > "${ssl_dir}/ca.cer"
+keytool -importcert -keystore "${ssl_dir}/ca.truststore" -alias ca
-storepass ${password} -noprompt -file "${ssl_dir}/ca.cer"
+
+keytool -genkeypair -alias node -keystore "${ssl_dir}/node.keystore"
-dname "CN=${NODENAME}" -ext SAN=${SANSTRING} -storepass ${password} -keypass
${password} -keyalg RSA -storetype PKCS12
+keytool -certreq -keystore "${ssl_dir}/node.keystore" -storepass
${password} -alias node -file "${ssl_dir}/node.csr"
+keytool -gencert -keystore "${ssl_dir}/ca.keystore" -storepass ${password}
-alias ca -ext SAN=${SANSTRING} -infile "${ssl_dir}/node.csr" -outfile
"${ssl_dir}/node.cer"
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+
+openssl pkcs12 -passin pass:${password} -in "${ssl_dir}/node.keystore"
-passout pass: -out "${ssl_dir}/node.pem" -nodes
Review comment:
where does this do and where does it come from? It doesn't documented in the
[SSL
docs](https://ci.apache.org/projects/flink/flink-docs-master/ops/security-ssl.html).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add end-to-end test for REST communication encryption
> -
>
> Key: FLINK-10628
> URL: https://issues.apache.org/jira/browse/FLINK-10628
> Project: Flink
> Issue Type: Sub-task
> Components: REST
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Andrey Zagrebin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> With FLINK-10371 we added support for mutual authentication for the REST
> communication. We should adapt one of the existing end-to-end tests to
> require this feature for the REST communication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[
https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671279#comment-16671279
]
ASF GitHub Bot commented on FLINK-10628:
zentol commented on a change in pull request #6967: [FLINK-10628][E2E][SSL]
Enable mutual REST SSL auth in e2e tests
URL: https://github.com/apache/flink/pull/6967#discussion_r229973793
##
File path: flink-end-to-end-tests/test-scripts/common_ssl.sh
##
@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+function _set_conf_ssl_helper {
+local type=$1 # 'internal' or external 'rest'
+local ssl_dir="${TEST_DATA_DIR}/ssl/${type}"
+local password="${type}.password"
+
+if [ "${type}" != "internal" ] && [ "${type}" != "rest" ]; then
+echo "Unknown type of ssl connectivity: ${type}. It can be either
'internal' or external 'rest'"
+exit 1
+fi
+
+# clean up the dir that will be used for SSL certificates and trust stores
+if [ -e "${ssl_dir}" ]; then
+ echo "File ${ssl_dir} exists. Deleting it..."
+ rm -rf "${ssl_dir}"
+fi
+mkdir -p "${ssl_dir}"
+
+SANSTRING="dns:${NODENAME}"
+for NODEIP in $(get_node_ip) ; do
+SANSTRING="${SANSTRING},ip:${NODEIP}"
+done
+
+echo "Using SAN ${SANSTRING}"
+
+# create certificates
+keytool -genkeypair -alias ca -keystore "${ssl_dir}/ca.keystore" -dname
"CN=Sample CA" -storepass ${password} -keypass ${password} -keyalg RSA -ext
bc=ca:true -storetype PKCS12
+keytool -keystore "${ssl_dir}/ca.keystore" -storepass ${password} -alias
ca -exportcert > "${ssl_dir}/ca.cer"
+keytool -importcert -keystore "${ssl_dir}/ca.truststore" -alias ca
-storepass ${password} -noprompt -file "${ssl_dir}/ca.cer"
+
+keytool -genkeypair -alias node -keystore "${ssl_dir}/node.keystore"
-dname "CN=${NODENAME}" -ext SAN=${SANSTRING} -storepass ${password} -keypass
${password} -keyalg RSA -storetype PKCS12
+keytool -certreq -keystore "${ssl_dir}/node.keystore" -storepass
${password} -alias node -file "${ssl_dir}/node.csr"
+keytool -gencert -keystore "${ssl_dir}/ca.keystore" -storepass ${password}
-alias ca -ext SAN=${SANSTRING} -infile "${ssl_dir}/node.csr" -outfile
"${ssl_dir}/node.cer"
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/ca.cer" -alias ca -noprompt
+keytool -importcert -keystore "${ssl_dir}/node.keystore" -storepass
${password} -file "${ssl_dir}/node.cer" -alias node -noprompt
+
+openssl pkcs12 -passin pass:${password} -in "${ssl_dir}/node.keystore"
-passout pass: -out "${ssl_dir}/node.pem" -nodes
+
+# adapt config
+# (here we rely on security.ssl.enabled enabling SSL for all components
and internal as well as
+# external communication channels)
+set_conf security.ssl.${type}.enabled true
+set_conf security.ssl.${type}.keystore ${ssl_dir}/node.keystore
+set_conf security.ssl.${type}.keystore-password ${password}
+set_conf security.ssl.${type}.key-password ${password}
+set_conf security.ssl.${type}.truststore ${ssl_dir}/ca.truststore
+set_conf security.ssl.${type}.truststore-password ${password}
+}
+
+function set_conf_internal_ssl {
+_set_conf_ssl_helper "internal"
+}
+
+function set_conf_mutual_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local mutual="false"
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+if [ "${auth}" == "mutual" ]; then
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cert ${ssl_dir}/node.pem"
+mutual="true";
+fi
+echo "Mutual ssl auth: ${mutual}"
+set_conf security.ssl.rest.authentication-enabled ${mutual}
+}
+
+function set_conf_rest_ssl {
+local auth="${1:-server}" # only 'server' or 'mutual'
+local ssl_dir="${TEST_DATA_DIR}/ssl/rest"
+_set_conf_ssl_helper "rest"
+set_conf_mutual_rest_ssl ${auth}
+REST_PROTOCOL="https"
+CURL_SSL_ARGS="${CURL_SSL_ARGS} --cacert
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[ https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669173#comment-16669173 ] ASF GitHub Bot commented on FLINK-10628: azagrebin opened a new pull request #6967: [FLINK-10628][E2E][SSL] Enable mutual REST SSL auth in e2e tests URL: https://github.com/apache/flink/pull/6967 ## What is the purpose of the change This PR enables mutual SSL authentication in batch all around and streaming file sink end-to-end tests. It also splits deprecated ssl configs into separate SSL configuration for internal and external REST communication. The curl commands, which query Flink REST API, are fixed with relevant SSL args to check and provide certificates where needed. ## Brief change log - separate common_ssl.sh from common.sh - fix curls with SSL in common.sh which query Flink - enable mutual SSL authentication in test_batch_allround.sh and test_streaming_file_sink.sh ## Verifying this change build Flink and run: FLINK_DIR=./build-target ./flink-end-to-end-tests/run-single-test.sh ./flink-end-to-end-tests/test-scripts/test_batch_allround.sh FLINK_DIR=./build-target ./flink-end-to-end-tests/run-single-test.sh ./flink-end-to-end-tests/test-scripts/test_streaming_file_sink.sh ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add end-to-end test for REST communication encryption > - > > Key: FLINK-10628 > URL: https://issues.apache.org/jira/browse/FLINK-10628 > Project: Flink > Issue Type: Sub-task > Components: REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Assignee: Andrey Zagrebin >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > With FLINK-10371 we added support for mutual authentication for the REST > communication. We should adapt one of the existing end-to-end tests to > require this feature for the REST communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[ https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658692#comment-16658692 ] Till Rohrmann commented on FLINK-10628: --- True, but I guess it won't hurt to change one of the existing e2e tests to use mutually authentication for the REST communication. > Add end-to-end test for REST communication encryption > - > > Key: FLINK-10628 > URL: https://issues.apache.org/jira/browse/FLINK-10628 > Project: Flink > Issue Type: Sub-task > Components: REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.7.0 > > > With FLINK-10371 we added support for mutual authentication for the REST > communication. We should adapt one of the existing end-to-end tests to > require this feature for the REST communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10628) Add end-to-end test for REST communication encryption
[ https://issues.apache.org/jira/browse/FLINK-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658669#comment-16658669 ] Chesnay Schepler commented on FLINK-10628: -- This is already covered by an IT case and seems redundant. > Add end-to-end test for REST communication encryption > - > > Key: FLINK-10628 > URL: https://issues.apache.org/jira/browse/FLINK-10628 > Project: Flink > Issue Type: Sub-task > Components: REST >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Major > Fix For: 1.7.0 > > > With FLINK-10371 we added support for mutual authentication for the REST > communication. We should adapt one of the existing end-to-end tests to > require this feature for the REST communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
