[ https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974621#comment-16974621 ]
Jeremy commented on SPARK-29884: -------------------------------- After doing some debugging it seams like this might be in fabric k8s client. I tries to use .kube/config even if it gets all the parameters is needs from arguments. > spark-submit to kuberentes can not parse valid ca certificate > ------------------------------------------------------------- > > Key: SPARK-29884 > URL: https://issues.apache.org/jira/browse/SPARK-29884 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 2.4.4 > Environment: A kuberentes cluster that has been in use for over 2 > years and handles large amounts of production payloads. > Reporter: Jeremy > Priority: Major > > spark submit can not be used to to schedule to kuberentes with oauth token > and cacert > {code:java} > spark-submit \ > --deploy-mode cluster \ > --class org.apache.spark.examples.SparkPi \ > --master k8s://https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com \ > --conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN \ > --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ > --conf > spark.kubernetes.authenticate.submission.caCertFile=/home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt > \ > --conf spark.kubernetes.namespace=here-olp-3dds-sit \ > --conf spark.executor.instances=1 \ > --conf spark.app.name=spark-pi \ > --conf > spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 > \ > --conf > spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 > \ > local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar > {code} > returns > {code:java} > log4j:WARN No appenders could be found for logger > (io.fabric8.kubernetes.client.Config). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > Exception in thread "main" > io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. > at > io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) > at > io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) > at > io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:183) > at > org.apache.spark.deploy.k8s.SparkKubernetesClientFactory$.createKubernetesClient(SparkKubernetesClientFactory.scala:84) > at > org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235) > at > org.apache.spark.deploy.k8s.submit.KubernetesClientApplication$$anonfun$run$4.apply(KubernetesClientApplication.scala:235) > at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2542) > at > org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:241) > at > org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:204) > at > org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845) > at > org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.security.cert.CertificateException: Could not parse > certificate: java.io.IOException: Empty input > at > sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110) > at > java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339) > at > io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:104) > at > io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:197) > at > io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128) > at > io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:122) > at > io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:78) > ... 13 more > Caused by: java.io.IOException: Empty input > at > sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:106) > ... 19 more > {code} > The cacert and token are both valid and work even with curl > {code:java} > curl --cacert /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt -H > "Authorization: bearer $TOKEN" -v > https://api.borg-dev-1-aws-eu-west-1.k8s.in.here.com/api/v1/namespaces/here-olp-3dds-sit/pods > -o out > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- > 0* Trying 10.117.233.37:443... > * TCP_NODELAY set > * Connected to api.borg-dev-1-aws-eu-west-1.k8s.in.here.com (10.117.233.37) > port 443 (#0) > * ALPN, offering h2 > * ALPN, offering http/1.1 > * successfully set certificate verify locations: > * CAfile: /home/jeremybr/.kube/borg-dev-1-aws-eu-west-1.crt > CApath: none > } [5 bytes data] > * TLSv1.3 (OUT), TLS handshake, Client hello (1): > } [512 bytes data] > * TLSv1.3 (IN), TLS handshake, Server hello (2): > { [58 bytes data] > * TLSv1.2 (IN), TLS handshake, Certificate (11): > { [1565 bytes data] > * TLSv1.2 (IN), TLS handshake, Server key exchange (12): > { [556 bytes data] > * TLSv1.2 (IN), TLS handshake, Request CERT (13): > { [588 bytes data] > * TLSv1.2 (IN), TLS handshake, Server finished (14): > { [4 bytes data] > * TLSv1.2 (OUT), TLS handshake, Certificate (11): > } [7 bytes data] > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): > } [37 bytes data] > * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): > } [1 bytes data] > * TLSv1.2 (OUT), TLS handshake, Finished (20): > } [16 bytes data] > * TLSv1.2 (IN), TLS handshake, Finished (20): > { [16 bytes data] > * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 > * ALPN, server accepted to use h2 > * Server certificate: > * subject: CN=kubernetes-master > * start date: Apr 2 13:30:06 2019 GMT > * expire date: Apr 1 13:30:06 2020 GMT > * subjectAltName: host "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com" > matched cert's "api.borg-dev-1-aws-eu-west-1.k8s.in.here.com" > * issuer: C=DE; ST=Berlin; O=HERE Global BV; OU=OLP Engineering > Infrastructure Development; CN=borg-dev-1-aws-eu-west-1; > emailAddress=sdp_ops_t...@here.com > * SSL certificate verify ok. > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- > 0* Using HTTP2, server supports multi-use > * Connection state changed (HTTP/2 confirmed) > * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: > len=0 > } [5 bytes data] > * Using Stream ID: 1 (easy handle 0x558a2ede87b0) > } [5 bytes data] > > GET /api/v1/namespaces/here-olp-3dds-sit/pods HTTP/2 > > Host: api.borg-dev-1-aws-eu-west-1.k8s.in.here.com > > User-Agent: curl/7.66.0 > > Accept: */* > > Authorization: bearer ***************************************** > > > { [5 bytes data] > * Connection state changed (MAX_CONCURRENT_STREAMS == 250)! > } [5 bytes data] > < HTTP/2 200 > < audit-id: a26869e9-7b42-4013-b1fa-839e963c6b78 > < content-type: application/json > < date: Wed, 13 Nov 2019 20:20:18 GMT > < > { [5 bytes data] > 100 56466 0 56466 0 0 64020 0 --:--:-- --:--:-- --:--:-- 63947 > * Connection #0 to host api.borg-dev-1-aws-eu-west-1.k8s.in.here.com left > intact > {code} > > {code:java} > // code placeholder > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org