something that would also help us narrow down the problematic area is to
enable SSL for one component at a time and see
which one causesd the job to fail.
On 04.10.2017 14:11, Chesnay Schepler wrote:
The configuration looks reasonable. Just to be sure, are the paths
accessible by all nodes?
As a first step, could you set the logging level to DEBUG (by
modifying the 'conf/log4j.properties' file), resubmit the job (after a
cluster restart) and check the Job- and TaskManager logs for any
exception?
On 04.10.2017 03:15, Aniket Deshpande wrote:
Background: We have a setup of Flink 1.3.1 along with a secure MAPR
cluster (Flink is running on mapr client nodes). We run this flink
cluster via flink-jobmanager.sh <http://flink-jobmanager.sh>
foreground and flink-taskmanager.sh <http://flink-taskmanager.sh>
foreground command via Marathon. In order for us to make this work,
we had to add -Djavax.net
<http://-Djavax.net>.ssl.trustStore="$JAVA_HOME/jre/lib/security/cacerts" in
flink-console.sh <http://flink-console.sh> as extra JVM arg
(otherwise, flink was taking MAPR's ssl_truststore as default
truststore and then we were facing issues for any 3rd party jars like
aws_sdk etc.). This entire setup was working fine as it is and we
could submit our jars and the pipelines ran without any problem
Problem: We started experimenting with enabling ssl for all
communication for Flink. For this, we followed
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/security-ssl.html for
generating CA and keystore. I added the following properties to
flink-conf.yaml:
security.ssl.enabled: true
security.ssl.keystore: /opt/flink/certs/node1.keystore
security.ssl.keystore-password: <password>
security.ssl.key-password: <password>
security.ssl.truststore: /opt/flink/certs/ca.truststore
security.ssl.truststore-password: <password>
jobmanager.web.ssl.enabled: true
taskmanager.data.ssl.enabled: true
blob.service.ssl.enabled: true
akka.ssl.enabled: true
We then spin up a cluster and tried submitting the same job which was
working before. We get the following erros:
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
load user class:
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer09
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.
at
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:229)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:95)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:230)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
This error disappears when we remove the ssl config properties i.e
run flink cluster without ssl enabled.
So, did we miss any steps for enabling ssl?
P.S.: We tried removing the extra JVm arg mentioned above, but still
get the same error.
--
Aniket
