todays
question.
From: Matt Cheah
Reply: Matt Cheah
Date: July 1, 2019 at 5:14:05 PM
To: Pat Ferrel ,
user@spark.apache.org
Subject: Re: k8s orchestrating Spark service
> We’d like to deploy Spark Workers/Executors and Master (whatever master
is easiest to talk about since we really do
then the ML server process has to create
a SparkContext object parameterized against the Kubernetes server in question.
I hope this helps!
-Matt Cheah
From: Pat Ferrel
Date: Monday, July 1, 2019 at 5:05 PM
To: "user@spark.apache.org" , Matt Cheah
Subject: Re: k8s orchestrating Spar
Subject: Re: k8s orchestrating Spark service
k8s as master would be nice but doesn’t solve the problem of running the
full cluster and is an orthogonal issue.
We’d like to deploy Spark Workers/Executors and Master (whatever master is
easiest to talk about since we really don’t care) in pods as we do
anyone have something they like?
From: Matt Cheah
Reply: Matt Cheah
Date: July 1, 2019 at 4:45:55 PM
To: Pat Ferrel ,
user@spark.apache.org
Subject: Re: k8s orchestrating Spark service
Sorry, I don’t quite follow – why use the Spark standalone cluster as an
in-between layer when one can just
ubject: Re: k8s orchestrating Spark service
Thanks Matt,
Actually I can’t use spark-submit. We submit the Driver programmatically
through the API. But this is not the issue and using k8s as the master is also
not the issue though you may be right about it being easier, it doesn’t quite
get to
of services including Spark. The rest work,
we are asking if anyone has seen a good starting point for adding Spark as
a k8s managed service.
From: Matt Cheah
Reply: Matt Cheah
Date: July 1, 2019 at 3:26:20 PM
To: Pat Ferrel ,
user@spark.apache.org
Subject: Re: k8s orchestrating Spark service
Date: Sunday, June 30, 2019 at 12:55 PM
To: "user@spark.apache.org"
Subject: k8s orchestrating Spark service
We're trying to setup a system that includes Spark. The rest of the services
have good Docker containers and Helm charts to start from.
Spark on the other hand
We're trying to setup a system that includes Spark. The rest of the
services have good Docker containers and Helm charts to start from.
Spark on the other hand is proving difficult. We forked a container and
have tried to create our own chart but are having several problems with
this.
So back to
