[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r227067136 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,43 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Client Deployment Mode + +To start STS in client mode, excute the following command + +```bash +$ sbin/start-thriftserver.sh \ +--master k8s://https://: +``` + +### Cluster Deployment Mode + +To start STS in cluster mode, excute the following command + +```bash +$ sbin/start-thriftserver.sh \ +--master k8s://https://: \ +--deploy-mode cluster +``` + +The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name(pod/container from --- End diff -- Yes, it might be rare to use this type of deployment on k8s, but it's still possible, e.g., for testing, to run the STS in client mode on a client machine. The wording here assumes that the server is running in a pod, which might not be the case. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
GitHub user suryag10 reopened a pull request: https://github.com/apache/spark/pull/22433 [SPARK-25442][SQL][K8S] Support STS to run in k8s deployments with spark deployment mode as cluster. ## What changes were proposed in this pull request? Code is enhanced to allow the STS run in kubernetes deployment with spark deploy mode of cluster. ## How was this patch tested? Started the sts in cluster mode in K8S deployment and was able to run some queries using the beeline client. You can merge this pull request into a Git repository by running: $ git pull https://github.com/suryag10/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22433.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22433 commit 3a7fa571181e4b0494f2b705fbd07bc61b0ca6ce Author: Suryanarayana GARLAPATI Date: 2018-09-16T04:37:26Z Support STS to run in k8s cluster mode commit 3556a61241e3f4910673f1bbf701905870ed09ea Author: Suryanarayana GARLAPATI Date: 2018-09-16T04:37:26Z [SPARK-25442][SQL][K8S] Support STS to run in k8s deployments with spark deployment mode as cluster. commit 78dc1a35f299d854b61c9b03e22730960c6280a2 Author: Suryanarayana GARLAPATI Date: 2018-09-19T05:16:48Z Merge branch 'master' of https://github.com/suryag10/spark commit a15f5313e7c798e58c80147e575218bb70fe2d74 Author: Suryanarayana GARLAPATI Date: 2018-09-19T05:16:48Z Merge branch 'master' of https://github.com/suryag10/spark commit 42dd479a33279b85fb9b3fa5a70570970e8148a1 Author: Suryanarayana GARLAPATI Date: 2018-09-19T17:08:58Z Merge branch 'master' of https://github.com/suryag10/spark commit 4a7e737ae1210451de668b13a72bbd9473721f45 Author: Suryanarayana GARLAPATI Date: 2018-09-19T17:08:58Z Merge branch 'master' of https://github.com/suryag10/spark commit d91fa2badf33cc4122d340af64ef669dddc66cf1 Author: Suryanarayana GARLAPATI Date: 2018-09-19T17:17:20Z Merge branch 'master' of https://github.com/suryag10/spark commit a65cfa56a64a7b9fb78a038ce6f9d25f2ce0e428 Author: Suryanarayana GARLAPATI Date: 2018-09-19T17:17:20Z Merge branch 'master' of https://github.com/suryag10/spark commit 8dc7ced8e44c1d75a27a15a9605d2bd8a693a732 Author: Suryanarayana GARLAPATI Date: 2018-09-19T17:23:03Z Merge branch 'master' of https://github.com/suryag10/spark commit 6e021e7706c12103ec9ce08b20d7fcb66c83aeb2 Author: Suryanarayana GARLAPATI Date: 2018-09-19T17:23:03Z Merge branch 'master' of https://github.com/suryag10/spark commit 12be1d2364e852b7cc27b78c0aec9740693e5cab Author: Suryanarayana GARLAPATI Date: 2018-09-21T12:33:12Z Merge branch 'master' of https://github.com/suryag10/spark --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user suryag10 closed the pull request at: https://github.com/apache/spark/pull/22433 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user suryag10 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r226898234 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,43 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Client Deployment Mode + +To start STS in client mode, excute the following command + +```bash +$ sbin/start-thriftserver.sh \ +--master k8s://https://: +``` + +### Cluster Deployment Mode + +To start STS in cluster mode, excute the following command + +```bash +$ sbin/start-thriftserver.sh \ +--master k8s://https://: \ +--deploy-mode cluster +``` + +The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name(pod/container from --- End diff -- STS is a server and its best way of deployment in K8S cluster is either done through the helm chart or through the yaml file(although it can be done through the method you had suggested, but i guess that scenario would be a rare case and there will be no HA of the STS server if it is triggered from outside). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r219728772 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,43 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Client Deployment Mode + +To start STS in client mode, excute the following command + +```bash +$ sbin/start-thriftserver.sh \ +--master k8s://https://: +``` + +### Cluster Deployment Mode + +To start STS in cluster mode, excute the following command + +```bash +$ sbin/start-thriftserver.sh \ +--master k8s://https://: \ +--deploy-mode cluster +``` + +The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name(pod/container from --- End diff -- The script may be run from a client machine outside a k8s cluster. In this case, there's not even a pod. I would suggest separating the explanation of the user flow details by the deploy mode (client vs cluster). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user suryag10 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r219476048 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Spark deploy mode of Client + +To start STS in client mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: + +### Spark deploy mode of Cluster + +To start STS in cluster mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: \ +--deploy-mode cluster + +The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name incase of client --- End diff -- pod/container from which the STS command is executed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r218950593 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Spark deploy mode of Client + +To start STS in client mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: + +### Spark deploy mode of Cluster + +To start STS in cluster mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: \ +--deploy-mode cluster + +The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name incase of client +mode, which can be found with kubectl get pods), and run kubectl port-forward spark-app-podname 31416:1 --- End diff -- Can we wrap commands into a pair of ``? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r218943664 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Spark deploy mode of Client + +To start STS in client mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: + +### Spark deploy mode of Cluster --- End diff -- Ditto, can we change this to `Cluster Deployment Mode`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r218950374 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Spark deploy mode of Client + +To start STS in client mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: + +### Spark deploy mode of Cluster + +To start STS in cluster mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: \ +--deploy-mode cluster + +The most basic workflow is to use the pod name (driver pod name incase of cluster mode and self pod name incase of client --- End diff -- What you mean by `self pod name`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r218944068 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Spark deploy mode of Client + +To start STS in client mode, excute the following command + +$ sbin/start-thriftserver.sh \ --- End diff -- Can we add put this into a code block? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r218944092 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Spark deploy mode of Client + +To start STS in client mode, excute the following command + +$ sbin/start-thriftserver.sh \ +--master k8s://https://: + +### Spark deploy mode of Cluster + +To start STS in cluster mode, excute the following command + +$ sbin/start-thriftserver.sh \ --- End diff -- Ditto. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22433: [SPARK-25442][SQL][K8S] Support STS to run in k8s...
Github user liyinan926 commented on a diff in the pull request: https://github.com/apache/spark/pull/22433#discussion_r218943584 --- Diff: docs/running-on-kubernetes.md --- @@ -340,6 +340,39 @@ RBAC authorization and how to configure Kubernetes service accounts for pods, pl [Using RBAC Authorization](https://kubernetes.io/docs/admin/authorization/rbac/) and [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/). +## Running Spark Thrift Server + +Thrift JDBC/ODBC Server (aka Spark Thrift Server or STS) is Spark SQLâs port of Apache Hiveâs HiveServer2 that allows +JDBC/ODBC clients to execute SQL queries over JDBC and ODBC protocols on Apache Spark. + +### Spark deploy mode of Client --- End diff -- Can we change the section title to just `Client Deployment Mode`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
