Hello, I have been trying to use Spark with Ignite (standalone mode) on a Kubernetes cluster and is running into some issues. Some details about the setup.
K8s Version: 1.9.7 Spark Version: 2.3.0 Ignite Version: 2.4.0 RBAC disabled. I have been using the example provided here<https://github.com/apache/ignite/blob/master/examples/src/main/spark/org/apache/ignite/examples/spark/SharedRDDExample.java>, only difference is that client mode is set to true. The stack trace is 2018-06-01 05:14:53 ERROR TcpDiscoverySpi:310 - Failed to get registered addresses from IP finder on start (retrying every 2000ms; change 'reconnectDelay' to configure the frequency of retries). class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses. at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1810) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1758) at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:494) at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$900(ClientImpl.java:125) at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:1837) at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1550) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) Caused by: java.io.FileNotFoundException: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/ignite at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1890) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263) at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153) ... 7 more I have explicitly set the masterUrl for the k8s in the config xml <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> <property name="ipFinder"> <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder"> <property name="namespace" value="default"/> <property name="serviceName" value="ignite-spark"/> <property name="masterUrl" value="https://k8s-apiserver.bcmt.cluster.local:8443"/> </bean> </property> </bean> but for some reason it is defaulting to the address https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/ignite . I have also attached the spark pod logs. I found a similar issue on the mailing list<http://apache-ignite-users.70518.x6.nabble.com/Unable-to-connect-ignite-pods-in-Kubernetes-using-Ip-finder-td18009.html> but RBAC is disabled on our k8s cluster. The spark submit command I used is as follows. bin/spark-submit --master k8s://https://k8s-apiserver.bcmt.cluster.local:8443 --deploy-mode cluster --name JavaIgniteRDDExample --class com.nokia.SharedRDDExample --conf spark.executor.instances=2 --conf spark.kubernetes.container.image=bcmt-registry:5000/spark:2.3 --jars <path/url to jars> <application_path/URL>
k8slogs
Description: k8slogs
