Hello Mich, Thank you for your questions. Here are my responses: > 1. What investigation have you done to show that it is running in local mode?
I have verified through the History Server's Environment tab that: - "spark.master" is set to local[*] - "spark.app.id" begins with local-xxx - "spark.submit.deployMode" is set to local > 2. who has configured this kubernetes cluster? Is it supplied by a cloud vendor? Our Kubernetes cluster was set up in an on-prem environment using RKE2( https://docs.rke2.io/ ). > 3. Confirm that you have configured Spark Connect Server correctly for cluster mode. Make sure you specify the cluster manager (e.g., Kubernetes) and other relevant Spark configurations in your Spark job submission. Based on the Spark Connect documentation I've read, there doesn't seem to be any specific settings for cluster mode related to the Spark Connect Server. Configuration - Spark 3.4.1 Documentation https://spark.apache.org/docs/3.4.1/configuration.html#spark-connect Quickstart: Spark Connect — PySpark 3.4.1 documentation https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_connect.html Spark Connect Overview - Spark 3.4.1 Documentation https://spark.apache.org/docs/latest/spark-connect-overview.html The documentation only suggests running ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.4.0, leaving me at a loss. > 4. Can you provide a full spark submit command Given the nature of Spark Connect, I don't use the spark-submit command. Instead, as per the documentation, I can execute workloads using only a Python script. For the Spark Connect Server, I have a Kubernetes manifest executing "/opt.spark/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.4.0". > 5. Make sure that the Python client script connecting to Spark Connect Server specifies the cluster mode explicitly, like using --master or --deploy-mode flags when creating a SparkSession. The Spark Connect Server operates as a Driver, so it isn't possible to specify the --master or --deploy-mode flags in the Python client script. If I try, I encounter a RuntimeError. like this: RuntimeError: Spark master cannot be configured with Spark Connect server; however, found URL for Spark Connect [sc://.../] > 6. Ensure that you have allocated the necessary resources (CPU, memory etc) to Spark Connect Server when running it on Kubernetes. Resources are ample, so that shouldn't be the problem. > 7. Review the environment variables and configurations you have set, including the SPARK_NO_DAEMONIZE=1 variable. Ensure that these variables are not conflicting with I'm unsure if SPARK_NO_DAEMONIZE=1 conflicts with cluster mode settings. But without it, the process goes to the background when executing start-connect-server.sh, causing the Pod to terminate prematurely. > 8. Are you using the correct spark client version that is fully compatible with your spark on the server? Yes, I have verified that without using Spark Connect (e.g., using Spark Operator), Spark applications run as expected. > 9. check the kubernetes error logs The Kubernetes logs don't show any errors, and jobs are running in local mode. > 10. Insufficient resources can lead to the application running in local mode I wasn't aware that insufficient resources could lead to local mode execution. Thank you for pointing it out. Best regards, Yasukazu 2023年9月5日(火) 1:28 Mich Talebzadeh <mich.talebza...@gmail.com>: > > personally I have not used this feature myself. However, some points > > > 1. What investigation have you done to show that it is running in > local mode? > 2. who has configured this kubernetes cluster? Is it supplied by a > cloud vendor? > 3. Confirm that you have configured Spark Connect Server correctly for > cluster mode. Make sure you specify the cluster manager (e.g., Kubernetes) > and other relevant Spark configurations in your Spark job submission. > 4. Can you provide a full spark submit command > 5. Make sure that the Python client script connecting to Spark Connect > Server specifies the cluster mode explicitly, like using --master or > --deploy-mode flags when creating a SparkSession. > 6. Ensure that you have allocated the necessary resources (CPU, memory > etc) to Spark Connect Server when running it on Kubernetes. > 7. Review the environment variables and configurations you have set, > including the SPARK_NO_DAEMONIZE=1 variable. Ensure that these > variables are not conflicting with cluster mode settings. > 8. Are you using the correct spark client version that is fully > compatible with your spark on the server? > 9. check the kubernetes error logs > 10. Insufficient resources can lead to the application running in > local mode > > HTH > > Mich Talebzadeh, > Distinguished Technologist, Solutions Architect & Engineer > London > United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 4 Sept 2023 at 04:57, Nagatomi Yasukazu <yassan0...@gmail.com> > wrote: > >> Hi Cley, >> >> Thank you for taking the time to respond to my query. Your insights on >> Spark cluster deployment are much appreciated. >> >> However, I'd like to clarify that my specific challenge is related to >> running the Spark Connect Server on Kubernetes in Cluster Mode. While I >> understand the general deployment strategies for Spark on Kubernetes, I am >> seeking guidance particularly on the Spark Connect Server aspect. >> >> cf. Spark Connect Overview - Spark 3.4.1 Documentation >> https://spark.apache.org/docs/latest/spark-connect-overview.html >> >> To reiterate, when I connect from an external Python client and execute >> scripts, the server operates in Local Mode instead of the expected >> Kubernetes Cluster Mode (with master as k8s://... and deploy-mode set to >> cluster). >> >> If I've misunderstood your initial response and it was indeed related to >> Spark Connect, I sincerely apologize for the oversight. In that case, could >> you please expand a bit on the Spark Connect-specific aspects? >> >> Do you, or anyone else in the community, have experience with this >> specific setup or encountered a similar issue with Spark Connect Server on >> Kubernetes? Any targeted advice or guidance would be invaluable. >> >> Thank you again for your time and help. >> >> Best regards, >> Yasukazu >> >> 2023年9月4日(月) 0:23 Cleyson Barros <euroc...@gmail.com>: >> >>> Hi Nagatomi, >>> Use Apache imagers, then run your master node, then start your many >>> slavers. You can add a command line in the docker files to call for the >>> master using the docker container names in your service composition if you >>> wish to run 2 masters active and standby follow the instructions in the >>> Apache docs to do this configuration, the recipe is the same except when >>> you start the masters and how you expect the behaviour of your cluster. >>> I hope it helps. >>> Have a nice day :) >>> Cley >>> >>> Nagatomi Yasukazu <yassan0...@gmail.com> escreveu no dia sábado, >>> 2/09/2023 à(s) 15:37: >>> >>>> Hello Apache Spark community, >>>> >>>> I'm currently trying to run Spark Connect Server on Kubernetes in >>>> Cluster Mode and facing some challenges. Any guidance or hints would be >>>> greatly appreciated. >>>> >>>> ## Environment: >>>> Apache Spark version: 3.4.1 >>>> Kubernetes version: 1.23 >>>> Command executed: >>>> /opt/spark/sbin/start-connect-server.sh \ >>>> --packages >>>> org.apache.spark:spark-connect_2.13:3.4.1,org.apache.iceberg:iceberg-spark-runtime-3.4_2.13:1.3.1... >>>> Note that I'm running it with the environment variable >>>> SPARK_NO_DAEMONIZE=1. >>>> >>>> ## Issue: >>>> When I connect from an external Python client and run scripts, it >>>> operates in Local Mode instead of the expected Cluster Mode. >>>> >>>> ## Expected Behavior: >>>> When connecting from a Python client to the Spark Connect Server, I >>>> expect it to run in Cluster Mode. >>>> >>>> If anyone has any insights, advice, or has faced a similar issue, I'd >>>> be grateful for your feedback. >>>> Thank you in advance. >>>> >>>> >>>>
