Github user ozzieba commented on the issue: https://github.com/apache/spark/pull/20272 @vanzin In general Kubernetes makes this super easy: - The most basic workflow is to use the driver pod name (which is in the output of Spark Submit, or can be found with `kubectl get pods`), and run `kubectl port-forward spark-app-driver-podname 31416:10000`, which will automatically [forward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) localhost:31416 to the pod's port 10000. Any jdbc application can then be used to query `jdbc:hive2://localhost:31416`. This is the approach I took in the integration tests linked above. - Alternatively, any other application on the cluster can simply use spark-app-driver-podname:10000, which will be resolved by [kube-dns](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/). - For persistent external access one can run `kubectl expose pod spark-app-driver-podname --type=NodePort --port 10000` to create a Kubernetes [Service](https://kubernetes.io/docs/concepts/services-networking/service/) which will accept connections on a particular port of *every* node on the cluster and send them to the driver's port 10000. - On a cloud environment, using `type=LoadBalancer` in the above will create a global [load balancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/) that can be used to access Thrift from across the Internet. - One can also define a service that automatically selects the driver using a user-specified label (eg `bin/spark-submit.sh --conf spark.kubernetes.driver.label.mythriftserver=true` followed by `kubectl expose pod --selector=mythriftserver=true`) Perhaps the key point is that these are all core Kubernetes features, and not specific to Spark in any way. Users familiar with Kubernetes should be able to find the approach that works best for their environment and use case.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org