Hi,
I personally do not use the Spark operator.
Anyhow, the Spark Operator automates the deployment and management of
Spark applications within Kubernetes. However, it does not eliminate the
need to configure Spark sessions for proper communication with the k8
cluster. So specifying the master URL is mandatory even when using the
Spark Operator in Kubernetes deployments.
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("AppConstants.APPLICATION_NAME") \
.config("spark.master",
"k8s://https://<kubernetes-api-server-address>:<port>")
\
.getOrCreate()
HTH
Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Thu, 4 Jan 2024 at 21:29, Atul Patil <atulsp...@gmail.com> wrote:
> Hello Team,
>
> I am currently working on initializing a Spark session using Spark
> Structure Streaming within a Kubernetes deployment managed by the Spark
> Operator. During the initialization process, I encountered an error message
> indicating the necessity to set a master URL:
>
> *"Caused by: org.apache.spark.SparkException: A master URL must be set in
> your configuration."*
>
> Could you kindly confirm if specifying the master URL is mandatory in this
> context of using Spark Structure Streaming programming with the Spark
> Operator for Kubernetes deployments?
>
> Below is the code snippet I am using for Spark session initialization:
> SparkSession sparkSession = SparkSession.builder()
> .appName(AppConstants.APPLICATION_NAME)
> .getOrCreate()
>
> Thank you!
>
> Regards,
> Atul
>
>
>
