[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r211285792
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyClientServerSslTest.java
##
@@ -98,21 +86,48 @@ public void testValidSslConnectionAdvanced() throws
Exception {
Channel ch = NettyTestUtil.connect(serverAndClient);
SslHandler sslHandler = (SslHandler) ch.pipeline().get("ssl");
- assertEquals(sslHandler.getHandshakeTimeoutMillis(),
handshakeTimeout);
- assertEquals(sslHandler.getCloseNotifyTimeoutMillis(),
closeNotifyFlushTimeout);
+ int handshakeTimeout =
sslConfig.getInteger(SSL_HANDSHAKE_TIMEOUT);
Review comment:
```
assertSslConfig(sslConfig.getInteger(SSL_HANDSHAKE_TIMEOUT),
sslHandler.getHandshakeTimeoutMillis())
```
and do:
```
assertSslConfig(expected, actual) {
if (expected != -1) {
assertEquals(expected, actual)
}
else {
assertTrue(...);
}
}
```
in 4 places here? Maybe renaming it to `assertEqualsOrMinusAsDefaultValue`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r210515177
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyServer.java
##
@@ -152,10 +154,17 @@ void init(final NettyProtocol protocol, NettyBufferPool
nettyBufferPool) throws
@Override
public void initChannel(SocketChannel channel) throws
Exception {
if (serverSSLContext != null) {
- SSLEngine sslEngine =
serverSSLContext.createSSLEngine();
+ SSLEngine sslEngine =
serverSSLContext.sslContext.createSSLEngine();
config.setSSLVerAndCipherSuites(sslEngine);
sslEngine.setUseClientMode(false);
- channel.pipeline().addLast("ssl", new
SslHandler(sslEngine));
+ SslHandler sslHandler = new
SslHandler(sslEngine);
Review comment:
`ctrl+v` - please deduplicate this somehow and please do this in this PR,
since this is the place where you introduce/make duplication worse.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r210515882
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java
##
@@ -176,39 +176,43 @@ public static void setSSLVerifyHostname(Configuration
sslConfig, SSLParameters s
public static SSLContext createSSLClientContext(Configuration
sslConfig) throws Exception {
Preconditions.checkNotNull(sslConfig);
- SSLContext clientSSLContext = null;
- if (getSSLEnabled(sslConfig)) {
- LOG.debug("Creating client SSL context from
configuration");
-
- String trustStoreFilePath =
sslConfig.getString(SecurityOptions.SSL_TRUSTSTORE);
- String trustStorePassword =
sslConfig.getString(SecurityOptions.SSL_TRUSTSTORE_PASSWORD);
- String sslProtocolVersion =
sslConfig.getString(SecurityOptions.SSL_PROTOCOL);
+ if (!getSSLEnabled(sslConfig)) {
+ return null;
+ }
- Preconditions.checkNotNull(trustStoreFilePath,
SecurityOptions.SSL_TRUSTSTORE.key() + " was not configured.");
- Preconditions.checkNotNull(trustStorePassword,
SecurityOptions.SSL_TRUSTSTORE_PASSWORD.key() + " was not configured.");
+ LOG.debug("Creating client SSL context from configuration");
- KeyStore trustStore =
KeyStore.getInstance(KeyStore.getDefaultType());
+ String trustStoreFilePath =
sslConfig.getString(SecurityOptions.SSL_TRUSTSTORE);
+ String trustStorePassword =
sslConfig.getString(SecurityOptions.SSL_TRUSTSTORE_PASSWORD);
+ String sslProtocolVersion =
sslConfig.getString(SecurityOptions.SSL_PROTOCOL);
+ int sessionCacheSize =
sslConfig.getInteger(SecurityOptions.SSL_SESSION_CACHE_SIZE);
+ int sessionTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_SESSION_TIMEOUT);
+ int handshakeTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_HANDSHAKE_TIMEOUT);
+ int closeNotifyFlushTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_CLOSE_NOTIFY_FLUSH_TIMEOUT);
- FileInputStream trustStoreFile = null;
- try {
- trustStoreFile = new FileInputStream(new
File(trustStoreFilePath));
- trustStore.load(trustStoreFile,
trustStorePassword.toCharArray());
- } finally {
- if (trustStoreFile != null) {
- trustStoreFile.close();
- }
- }
+ Preconditions.checkNotNull(trustStoreFilePath,
SecurityOptions.SSL_TRUSTSTORE.key() + " was not configured.");
+ Preconditions.checkNotNull(trustStorePassword,
SecurityOptions.SSL_TRUSTSTORE_PASSWORD.key() + " was not configured.");
- TrustManagerFactory trustManagerFactory =
TrustManagerFactory.getInstance(
- TrustManagerFactory.getDefaultAlgorithm());
- trustManagerFactory.init(trustStore);
+ KeyStore trustStore =
KeyStore.getInstance(KeyStore.getDefaultType());
- clientSSLContext =
SSLContext.getInstance(sslProtocolVersion);
- clientSSLContext.init(null,
trustManagerFactory.getTrustManagers(), null);
+ try (FileInputStream trustStoreFile = new FileInputStream(new
File(trustStoreFilePath))) {
+ trustStore.load(trustStoreFile,
trustStorePassword.toCharArray());
}
- return clientSSLContext;
+ TrustManagerFactory trustManagerFactory =
TrustManagerFactory.getInstance(
+ TrustManagerFactory.getDefaultAlgorithm());
+ trustManagerFactory.init(trustStore);
+
+ javax.net.ssl.SSLContext clientSSLContext =
javax.net.ssl.SSLContext.getInstance(sslProtocolVersion);
Review comment:
`ctrl+c`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r210523907
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java
##
@@ -225,38 +229,65 @@ public static SSLContext
createSSLClientContext(Configuration sslConfig) throws
public static SSLContext createSSLServerContext(Configuration
sslConfig) throws Exception {
Preconditions.checkNotNull(sslConfig);
- SSLContext serverSSLContext = null;
- if (getSSLEnabled(sslConfig)) {
- LOG.debug("Creating server SSL context from
configuration");
+ if (!getSSLEnabled(sslConfig)) {
+ return null;
+ }
- String keystoreFilePath =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE);
+ LOG.debug("Creating server SSL context from configuration");
- String keystorePassword =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE_PASSWORD);
+ String keystoreFilePath =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE);
+ String keystorePassword =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE_PASSWORD);
+ String certPassword =
sslConfig.getString(SecurityOptions.SSL_KEY_PASSWORD);
+ String sslProtocolVersion =
sslConfig.getString(SecurityOptions.SSL_PROTOCOL);
+ int sessionCacheSize =
sslConfig.getInteger(SecurityOptions.SSL_SESSION_CACHE_SIZE);
+ int sessionTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_SESSION_TIMEOUT);
+ int handshakeTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_HANDSHAKE_TIMEOUT);
+ int closeNotifyFlushTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_CLOSE_NOTIFY_FLUSH_TIMEOUT);
- String certPassword =
sslConfig.getString(SecurityOptions.SSL_KEY_PASSWORD);
+ Preconditions.checkNotNull(keystoreFilePath,
SecurityOptions.SSL_KEYSTORE.key() + " was not configured.");
+ Preconditions.checkNotNull(keystorePassword,
SecurityOptions.SSL_KEYSTORE_PASSWORD.key() + " was not configured.");
+ Preconditions.checkNotNull(certPassword,
SecurityOptions.SSL_KEY_PASSWORD.key() + " was not configured.");
- String sslProtocolVersion =
sslConfig.getString(SecurityOptions.SSL_PROTOCOL);
+ KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
+ try (FileInputStream keyStoreFile = new FileInputStream(new
File(keystoreFilePath))) {
+ ks.load(keyStoreFile, keystorePassword.toCharArray());
+ }
- Preconditions.checkNotNull(keystoreFilePath,
SecurityOptions.SSL_KEYSTORE.key() + " was not configured.");
- Preconditions.checkNotNull(keystorePassword,
SecurityOptions.SSL_KEYSTORE_PASSWORD.key() + " was not configured.");
- Preconditions.checkNotNull(certPassword,
SecurityOptions.SSL_KEY_PASSWORD.key() + " was not configured.");
+ // Set up key manager factory to use the server key store
+ KeyManagerFactory kmf = KeyManagerFactory.getInstance(
+ KeyManagerFactory.getDefaultAlgorithm());
+ kmf.init(ks, certPassword.toCharArray());
- KeyStore ks =
KeyStore.getInstance(KeyStore.getDefaultType());
- try (FileInputStream keyStoreFile = new
FileInputStream(new File(keystoreFilePath))) {
- ks.load(keyStoreFile,
keystorePassword.toCharArray());
- }
+ // Initialize the SSLContext
+ javax.net.ssl.SSLContext serverSSLContext =
javax.net.ssl.SSLContext.getInstance(sslProtocolVersion);
+ serverSSLContext.init(kmf.getKeyManagers(), null, null);
+ if (sessionCacheSize >= 0) {
+
serverSSLContext.getServerSessionContext().setSessionCacheSize(sessionCacheSize);
+ }
+ if (sessionTimeoutMs >= 0) {
+
serverSSLContext.getServerSessionContext().setSessionTimeout(sessionTimeoutMs /
1000);
+ }
- // Set up key manager factory to use the server key
store
- KeyManagerFactory kmf = KeyManagerFactory.getInstance(
-
KeyManagerFactory.getDefaultAlgorithm());
- kmf.init(ks, certPassword.toCharArray());
+ return new SSLContext(serverSSLContext, handshakeTimeoutMs,
closeNotifyFlushTimeoutMs);
+ }
- // Initialize the SSLContext
- serverSSLContext =
SSLContext.getInstance(sslProtocolVersion);
- serv
[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r210529443
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyClientServerSslTest.java
##
@@ -65,6 +68,60 @@ public void testValidSslConnection() throws Exception {
Channel ch = NettyTestUtil.connect(serverAndClient);
+ SslHandler sslHandler = (SslHandler) ch.pipeline().get("ssl");
+ assertTrue("default value should not be propagated",
sslHandler.getHandshakeTimeoutMillis() >= 0);
+ assertTrue("default value should not be propagated",
sslHandler.getCloseNotifyTimeoutMillis() >= 0);
+
+ // should be able to send text data
+ ch.pipeline().addLast(new StringDecoder()).addLast(new
StringEncoder());
+ assertTrue(ch.writeAndFlush("test").await().isSuccess());
+
+ NettyTestUtil.shutdown(serverAndClient);
+ }
+
+ /**
+* Verify valid (advanced) ssl configuration and connection.
+*/
+ @Test
+ public void testValidSslConnectionAdvanced() throws Exception {
Review comment:
This hasn't been addressed. Those tests differ only with expected values and
passed `NettyConfig`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r210514431
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyClient.java
##
@@ -190,7 +194,14 @@ public void initChannel(SocketChannel channel) throws
Exception {
sslEngine.setSSLParameters(newSSLParameters);
}
- channel.pipeline().addLast("ssl", new
SslHandler(sslEngine));
+ SslHandler sslHandler = new
SslHandler(sslEngine);
Review comment:
`ctrl+c`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r210515985
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/net/SSLUtils.java
##
@@ -225,38 +229,65 @@ public static SSLContext
createSSLClientContext(Configuration sslConfig) throws
public static SSLContext createSSLServerContext(Configuration
sslConfig) throws Exception {
Preconditions.checkNotNull(sslConfig);
- SSLContext serverSSLContext = null;
- if (getSSLEnabled(sslConfig)) {
- LOG.debug("Creating server SSL context from
configuration");
+ if (!getSSLEnabled(sslConfig)) {
+ return null;
+ }
- String keystoreFilePath =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE);
+ LOG.debug("Creating server SSL context from configuration");
- String keystorePassword =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE_PASSWORD);
+ String keystoreFilePath =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE);
+ String keystorePassword =
sslConfig.getString(SecurityOptions.SSL_KEYSTORE_PASSWORD);
+ String certPassword =
sslConfig.getString(SecurityOptions.SSL_KEY_PASSWORD);
+ String sslProtocolVersion =
sslConfig.getString(SecurityOptions.SSL_PROTOCOL);
+ int sessionCacheSize =
sslConfig.getInteger(SecurityOptions.SSL_SESSION_CACHE_SIZE);
+ int sessionTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_SESSION_TIMEOUT);
+ int handshakeTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_HANDSHAKE_TIMEOUT);
+ int closeNotifyFlushTimeoutMs =
sslConfig.getInteger(SecurityOptions.SSL_CLOSE_NOTIFY_FLUSH_TIMEOUT);
- String certPassword =
sslConfig.getString(SecurityOptions.SSL_KEY_PASSWORD);
+ Preconditions.checkNotNull(keystoreFilePath,
SecurityOptions.SSL_KEYSTORE.key() + " was not configured.");
+ Preconditions.checkNotNull(keystorePassword,
SecurityOptions.SSL_KEYSTORE_PASSWORD.key() + " was not configured.");
+ Preconditions.checkNotNull(certPassword,
SecurityOptions.SSL_KEY_PASSWORD.key() + " was not configured.");
- String sslProtocolVersion =
sslConfig.getString(SecurityOptions.SSL_PROTOCOL);
+ KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
+ try (FileInputStream keyStoreFile = new FileInputStream(new
File(keystoreFilePath))) {
+ ks.load(keyStoreFile, keystorePassword.toCharArray());
+ }
- Preconditions.checkNotNull(keystoreFilePath,
SecurityOptions.SSL_KEYSTORE.key() + " was not configured.");
- Preconditions.checkNotNull(keystorePassword,
SecurityOptions.SSL_KEYSTORE_PASSWORD.key() + " was not configured.");
- Preconditions.checkNotNull(certPassword,
SecurityOptions.SSL_KEY_PASSWORD.key() + " was not configured.");
+ // Set up key manager factory to use the server key store
+ KeyManagerFactory kmf = KeyManagerFactory.getInstance(
+ KeyManagerFactory.getDefaultAlgorithm());
+ kmf.init(ks, certPassword.toCharArray());
- KeyStore ks =
KeyStore.getInstance(KeyStore.getDefaultType());
- try (FileInputStream keyStoreFile = new
FileInputStream(new File(keystoreFilePath))) {
- ks.load(keyStoreFile,
keystorePassword.toCharArray());
- }
+ // Initialize the SSLContext
Review comment:
`ctrl+v` as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] pnowojski commented on a change in pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl options
pnowojski commented on a change in pull request #6355:
[FLINK-9878][network][ssl] add more low-level ssl options
URL: https://github.com/apache/flink/pull/6355#discussion_r210531098
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyClientServerSslTest.java
##
@@ -65,6 +68,60 @@ public void testValidSslConnection() throws Exception {
Channel ch = NettyTestUtil.connect(serverAndClient);
+ SslHandler sslHandler = (SslHandler) ch.pipeline().get("ssl");
+ assertTrue("default value should not be propagated",
sslHandler.getHandshakeTimeoutMillis() >= 0);
+ assertTrue("default value should not be propagated",
sslHandler.getCloseNotifyTimeoutMillis() >= 0);
+
+ // should be able to send text data
+ ch.pipeline().addLast(new StringDecoder()).addLast(new
StringEncoder());
+ assertTrue(ch.writeAndFlush("test").await().isSuccess());
+
+ NettyTestUtil.shutdown(serverAndClient);
+ }
+
+ /**
+* Verify valid (advanced) ssl configuration and connection.
+*/
+ @Test
+ public void testValidSslConnectionAdvanced() throws Exception {
Review comment:
Yes, you are right regarding `handshake-timeout` and
`close-notify-flush-timeout`, but as I wrote previously, I do not see how
`SESSION_CACHE_SIZE` and `SESSION_TIMEOUT` are tested at all. And regardless of
that, it still would be better to add a stress test/benchmark for that. Depends
how important this feature is... However if it's not important one could argue
why even bother supporting this?
On a side note. Couldn't we provide some generic way to configure netty?
Like passing any config option prefixed `taskmanager.netty.client` to
taskmanager's netty client, without manually specifying and handling them by us?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
