Re: Kubernetes backend and docker images

[Anirudh Ramanathan]

+1

We discussed some alternatives early on - including using a single
dockerfile and different spec.container.command and spec.container.args
from the Kubernetes driver/executor specification (which override
entrypoint in docker). No reason that won't work also - except that it
reduced the transparency of what was being invoked in the
driver/executor/init by hiding it in the actual backend code.

Putting it into a single entrypoint file and branching let's us realize the
best of both worlds I think. This is an elegant solution, thanks Marcelo.

On Jan 6, 2018 10:01 AM, "Felix Cheung"  wrote:

> +1
>
> Thanks for taking on this.
> That was my feedback on one of the long comment thread as well, I think we
> should have one docker image instead of 3 (also pending in the fork are
> python and R variant, we should consider having one that we official
> release instead of 9, for example)
>
>
> --
> *From:* 蒋星博 
> *Sent:* Friday, January 5, 2018 10:57:53 PM
> *To:* Marcelo Vanzin
> *Cc:* dev
> *Subject:* Re: Kubernetes backend and docker images
>
> Agree it should be nice to have this simplification, and users can still
> create their custom images by copy/modifying the default one.
> Thanks for bring this out Marcelo!
>
> 2018-01-05 17:06 GMT-08:00 Marcelo Vanzin :
>
>> Hey all, especially those working on the k8s stuff.
>>
>> Currently we have 3 docker images that need to be built and provided
>> by the user when starting a Spark app: driver, executor, and init
>> container.
>>
>> When the initial review went by, I asked why do we need 3, and I was
>> told that's because they have different entry points. That never
>> really convinced me, but well, everybody wanted to get things in to
>> get the ball rolling.
>>
>> But I still think that's not the best way to go. I did some pretty
>> simple hacking and got things to work with a single image:
>>
>> https://github.com/vanzin/spark/commit/k8s-img
>>
>> Is there a reason why that approach would not work? You could still
>> create separate images for driver and executor if wanted, but there's
>> no reason I can see why we should need 3 images for the simple case.
>>
>> Note that the code there can be cleaned up still, and I don't love the
>> idea of using env variables to propagate arguments to the container,
>> but that works for now.
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>

Re: Kubernetes backend and docker images

[Anirudh Ramanathan]

+matt +tim
For reference - here's our previous thread on this dockerfile unification
problem - https://github.com/apache-spark-on-k8s/spark/pull/60
I think this approach should be acceptable from both the customization and
visibility perspectives.


On Mon, Jan 8, 2018 at 9:40 AM, Anirudh Ramanathan 
wrote:

> +1
>
> We discussed some alternatives early on - including using a single
> dockerfile and different spec.container.command and spec.container.args
> from the Kubernetes driver/executor specification (which override
> entrypoint in docker). No reason that won't work also - except that it
> reduced the transparency of what was being invoked in the
> driver/executor/init by hiding it in the actual backend code.
>
> Putting it into a single entrypoint file and branching let's us realize
> the best of both worlds I think. This is an elegant solution, thanks
> Marcelo.
>
> On Jan 6, 2018 10:01 AM, "Felix Cheung"  wrote:
>
>> +1
>>
>> Thanks for taking on this.
>> That was my feedback on one of the long comment thread as well, I think
>> we should have one docker image instead of 3 (also pending in the fork are
>> python and R variant, we should consider having one that we official
>> release instead of 9, for example)
>>
>>
>> --
>> *From:* 蒋星博 
>> *Sent:* Friday, January 5, 2018 10:57:53 PM
>> *To:* Marcelo Vanzin
>> *Cc:* dev
>> *Subject:* Re: Kubernetes backend and docker images
>>
>> Agree it should be nice to have this simplification, and users can still
>> create their custom images by copy/modifying the default one.
>> Thanks for bring this out Marcelo!
>>
>> 2018-01-05 17:06 GMT-08:00 Marcelo Vanzin :
>>
>>> Hey all, especially those working on the k8s stuff.
>>>
>>> Currently we have 3 docker images that need to be built and provided
>>> by the user when starting a Spark app: driver, executor, and init
>>> container.
>>>
>>> When the initial review went by, I asked why do we need 3, and I was
>>> told that's because they have different entry points. That never
>>> really convinced me, but well, everybody wanted to get things in to
>>> get the ball rolling.
>>>
>>> But I still think that's not the best way to go. I did some pretty
>>> simple hacking and got things to work with a single image:
>>>
>>> https://github.com/vanzin/spark/commit/k8s-img
>>>
>>> Is there a reason why that approach would not work? You could still
>>> create separate images for driver and executor if wanted, but there's
>>> no reason I can see why we should need 3 images for the simple case.
>>>
>>> Note that the code there can be cleaned up still, and I don't love the
>>> idea of using env variables to propagate arguments to the container,
>>> but that works for now.
>>>
>>> --
>>> Marcelo
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>


-- 
Anirudh Ramanathan

Re: Kubernetes backend and docker images

[Matt Cheah]

We would still want images to be able to be uniquely specified for the driver 
vs. the executors. For example, not all of the libraries required on the driver 
may be required on the executors, so the user would want to specify a different 
custom driver image from their custom executor image.

 

But the idea of the entry point script that can switch based on environment 
variables makes sense.

 

I do think we want separate Python and R images, because Python and R come with 
non-trivial extra baggage that can make the images a lot bigger and slower to 
download for Scala-only users.

 

From: Anirudh Ramanathan 
Date: Monday, January 8, 2018 at 9:48 AM
To: Felix Cheung 
Cc: 蒋星博 , Marcelo Vanzin , dev 
, Matt Cheah , Timothy Chen 

Subject: Re: Kubernetes backend and docker images

 

+matt +tim

For reference - here's our previous thread on this dockerfile unification 
problem - https://github.com/apache-spark-on-k8s/spark/pull/60[github.com] 

I think this approach should be acceptable from both the customization and 
visibility perspectives.

 

 

On Mon, Jan 8, 2018 at 9:40 AM, Anirudh Ramanathan  
wrote:

+1

 

We discussed some alternatives early on - including using a single dockerfile 
and different spec.container.command and spec.container.args from the 
Kubernetes driver/executor specification (which override entrypoint in docker). 
No reason that won't work also - except that it reduced the transparency of 
what was being invoked in the driver/executor/init by hiding it in the actual 
backend code.   

 

Putting it into a single entrypoint file and branching let's us realize the 
best of both worlds I think. This is an elegant solution, thanks Marcelo. 

 

On Jan 6, 2018 10:01 AM, "Felix Cheung"  wrote:

+1

 

Thanks for taking on this.

That was my feedback on one of the long comment thread as well, I think we 
should have one docker image instead of 3 (also pending in the fork are python 
and R variant, we should consider having one that we official release instead 
of 9, for example)

 

 

From: 蒋星博 
Sent: Friday, January 5, 2018 10:57:53 PM
To: Marcelo Vanzin
Cc: dev
Subject: Re: Kubernetes backend and docker images 

 

Agree it should be nice to have this simplification, and users can still create 
their custom images by copy/modifying the default one. 

Thanks for bring this out Marcelo!

 

2018-01-05 17:06 GMT-08:00 Marcelo Vanzin :

Hey all, especially those working on the k8s stuff.

Currently we have 3 docker images that need to be built and provided
by the user when starting a Spark app: driver, executor, and init
container.

When the initial review went by, I asked why do we need 3, and I was
told that's because they have different entry points. That never
really convinced me, but well, everybody wanted to get things in to
get the ball rolling.

But I still think that's not the best way to go. I did some pretty
simple hacking and got things to work with a single image:

https://github.com/vanzin/spark/commit/k8s-img[github.com]

Is there a reason why that approach would not work? You could still
create separate images for driver and executor if wanted, but there's
no reason I can see why we should need 3 images for the simple case.

Note that the code there can be cleaned up still, and I don't love the
idea of using env variables to propagate arguments to the container,
but that works for now.

--
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

 



 

-- 

Anirudh Ramanathan



smime.p7s
Description: S/MIME cryptographic signature

Re: Kubernetes backend and docker images

[Matt Cheah]

// Fixing Anirudh's email address


From: Matt Cheah
Sent: Monday, January 8, 2018 1:39:12 PM
To: Anirudh Ramanathan; Felix Cheung
Cc: 蒋星博; Marcelo Vanzin; dev; Timothy Chen
Subject: Re: Kubernetes backend and docker images


We would still want images to be able to be uniquely specified for the driver 
vs. the executors. For example, not all of the libraries required on the driver 
may be required on the executors, so the user would want to specify a different 
custom driver image from their custom executor image.



But the idea of the entry point script that can switch based on environment 
variables makes sense.



I do think we want separate Python and R images, because Python and R come with 
non-trivial extra baggage that can make the images a lot bigger and slower to 
download for Scala-only users.



From: Anirudh Ramanathan 
Date: Monday, January 8, 2018 at 9:48 AM
To: Felix Cheung 
Cc: 蒋星博 , Marcelo Vanzin , dev 
, Matt Cheah , Timothy Chen 

Subject: Re: Kubernetes backend and docker images



+matt +tim

For reference - here's our previous thread on this dockerfile unification 
problem - 
https://github.com/apache-spark-on-k8s/spark/pull/60[github.com]

I think this approach should be acceptable from both the customization and 
visibility perspectives.





On Mon, Jan 8, 2018 at 9:40 AM, Anirudh Ramanathan 
mailto:ramanath...@google.com>> wrote:

+1



We discussed some alternatives early on - including using a single dockerfile 
and different spec.container.command and spec.container.args from the 
Kubernetes driver/executor specification (which override entrypoint in docker). 
No reason that won't work also - except that it reduced the transparency of 
what was being invoked in the driver/executor/init by hiding it in the actual 
backend code.



Putting it into a single entrypoint file and branching let's us realize the 
best of both worlds I think. This is an elegant solution, thanks Marcelo.



On Jan 6, 2018 10:01 AM, "Felix Cheung" 
mailto:felixcheun...@hotmail.com>> wrote:

+1



Thanks for taking on this.

That was my feedback on one of the long comment thread as well, I think we 
should have one docker image instead of 3 (also pending in the fork are python 
and R variant, we should consider having one that we official release instead 
of 9, for example)







From: 蒋星博 mailto:jiangxb1...@gmail.com>>
Sent: Friday, January 5, 2018 10:57:53 PM
To: Marcelo Vanzin
Cc: dev
Subject: Re: Kubernetes backend and docker images



Agree it should be nice to have this simplification, and users can still create 
their custom images by copy/modifying the default one.

Thanks for bring this out Marcelo!



2018-01-05 17:06 GMT-08:00 Marcelo Vanzin 
mailto:van...@cloudera.com>>:

Hey all, especially those working on the k8s stuff.

Currently we have 3 docker images that need to be built and provided
by the user when starting a Spark app: driver, executor, and init
container.

When the initial review went by, I asked why do we need 3, and I was
told that's because they have different entry points. That never
really convinced me, but well, everybody wanted to get things in to
get the ball rolling.

But I still think that's not the best way to go. I did some pretty
simple hacking and got things to work with a single image:

https://github.com/vanzin/spark/commit/k8s-img[github.com]

Is there a reason why that approach would not work? You could still
create separate images for driver and executor if wanted, but there's
no reason I can see why we should need 3 images for the simple case.

Note that the code there can be cleaned up still, and I don't love the
idea of using env variables to propagate arguments to the container,
but that works for now.

--
Marcelo

-
To unsubscribe e-mail: 
dev-unsubscr...@spark.apache.org







--

Anirudh Ramanathan

Re: Kubernetes backend and docker images

[Marcelo Vanzin]

On Mon, Jan 8, 2018 at 1:39 PM, Matt Cheah  wrote:
> We would still want images to be able to be uniquely specified for the
> driver vs. the executors. For example, not all of the libraries required on
> the driver may be required on the executors, so the user would want to
> specify a different custom driver image from their custom executor image.

Are you saying that we should *require* different images for driver
and executor, as is the case today, or that we should *allow*
different images, but default to the same, as I'm proposing?

I see zero reason to require different images. While it's true that
the driver may need more libraries than the executor, 99% of the time
it's ok to just have those libraries everywhere - it makes
configuration easier and doesn't do any harm.


-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Kubernetes backend and docker images

[Matt Cheah]

Think we can allow for different images and default to them being the same. 
Apologize if I missed that as being the original intention though.

-Matt Cheah

On 1/8/18, 1:45 PM, "Marcelo Vanzin"  wrote:

On Mon, Jan 8, 2018 at 1:39 PM, Matt Cheah  wrote:
> We would still want images to be able to be uniquely specified for the
> driver vs. the executors. For example, not all of the libraries required 
on
> the driver may be required on the executors, so the user would want to
> specify a different custom driver image from their custom executor image.

Are you saying that we should *require* different images for driver
and executor, as is the case today, or that we should *allow*
different images, but default to the same, as I'm proposing?

I see zero reason to require different images. While it's true that
the driver may need more libraries than the executor, 99% of the time
it's ok to just have those libraries everywhere - it makes
configuration easier and doesn't do any harm.


-- 
Marcelo



smime.p7s
Description: S/MIME cryptographic signature

Re: Branch 2.3 is cut

[Sameer Agarwal]

Hello everyone,

Just a quick update on the release. There are currently 2 correctness
blockers (SPARK-22984 
 and SPARK-22982 ) that
are targeted against 2.3.0. We'll go ahead and create an RC as soon as
they're resolved. All relevant jenkins jobs for the release branch can be
accessed at: https://amplab.cs.berkeley.edu/jenkins/

Regards,
Sameer

On Mon, Jan 1, 2018 at 5:22 PM, Sameer Agarwal 
wrote:

> We've just cut the release branch for Spark 2.3. Committers, please
> backport all important bug fixes and PRs as appropriate.
>
> Next, I'll go ahead and create the jenkins jobs for the release branch and
> then follow up with an RC early next week.
>



-- 
Sameer Agarwal
Software Engineer | Databricks Inc.
http://cs.berkeley.edu/~sameerag

Integration testing and Scheduler Backends

[Anirudh Ramanathan]

This is with regard to the Kubernetes Scheduler Backend and scaling the
process to accept contributions. Given we're moving past upstreaming
changes from our fork, and into getting *new* patches, I wanted to start
this discussion sooner than later. This is more of a post-2.3 question -
not something we're looking to solve right away.

While unit tests are handy, they're not nearly as good at giving us
confidence as a successful run of our integration tests against
single/multi-node k8s clusters. Currently, we have integration testing
setup at https://github.com/apache-spark-on-k8s/spark-integration and it's
running continuously against apache/spark:master in pepperdata-jenkins
 (on
minikube) & k8s-testgrid
 (in
GKE clusters). Now, the question is - how do we make integration-tests part
of the PR author's workflow?

1. Keep the integration tests in the separate repo and require that
contributors run them, add new tests prior to accepting their PRs as a
policy. Given minikube  is easy to
setup and can run on a single-node, it would certainly be possible.
Friction however, stems from contributors potentially having to modify the
integration test code hosted in that separate repository when
adding/changing functionality in the scheduler backend. Also, it's
certainly going to lead to at least brief inconsistencies between the two
repositories.

2. Alternatively, we check in the integration tests alongside the actual
scheduler backend code. This would work really well and is what we did in
our fork. It would have to be a separate package which would take certain
parameters (like cluster endpoint) and run integration test code against a
local or remote cluster. It would include least some code dealing with
accessing the cluster, reading results from K8s containers, test fixtures,
etc.

I see value in adopting (2), given it's a clearer path for contributors and
lets us keep the two pieces consistent, but it seems uncommon elsewhere.
How do the other backends, i.e. YARN, Mesos and Standalone deal with
accepting patches and ensuring that they do not break existing clusters? Is
there automation employed for this thus far? Would love to get opinions on
(1) v/s (2).

Thanks,
Anirudh

Re: Integration testing and Scheduler Backends

[Felix Cheung]

How would (2) be uncommon elsewhere?

On Mon, Jan 8, 2018 at 10:16 PM Anirudh Ramanathan 
wrote:

> This is with regard to the Kubernetes Scheduler Backend and scaling the
> process to accept contributions. Given we're moving past upstreaming
> changes from our fork, and into getting *new* patches, I wanted to start
> this discussion sooner than later. This is more of a post-2.3 question -
> not something we're looking to solve right away.
>
> While unit tests are handy, they're not nearly as good at giving us
> confidence as a successful run of our integration tests against
> single/multi-node k8s clusters. Currently, we have integration testing
> setup at https://github.com/apache-spark-on-k8s/spark-integration and
> it's running continuously against apache/spark:master in
> pepperdata-jenkins
>  (on
> minikube) & k8s-testgrid
>  (in
> GKE clusters). Now, the question is - how do we make integration-tests
> part of the PR author's workflow?
>
> 1. Keep the integration tests in the separate repo and require that
> contributors run them, add new tests prior to accepting their PRs as a
> policy. Given minikube  is easy
> to setup and can run on a single-node, it would certainly be possible.
> Friction however, stems from contributors potentially having to modify the
> integration test code hosted in that separate repository when
> adding/changing functionality in the scheduler backend. Also, it's
> certainly going to lead to at least brief inconsistencies between the two
> repositories.
>
> 2. Alternatively, we check in the integration tests alongside the actual
> scheduler backend code. This would work really well and is what we did in
> our fork. It would have to be a separate package which would take certain
> parameters (like cluster endpoint) and run integration test code against a
> local or remote cluster. It would include least some code dealing with
> accessing the cluster, reading results from K8s containers, test fixtures,
> etc.
>
> I see value in adopting (2), given it's a clearer path for contributors
> and lets us keep the two pieces consistent, but it seems uncommon
> elsewhere. How do the other backends, i.e. YARN, Mesos and Standalone deal
> with accepting patches and ensuring that they do not break existing
> clusters? Is there automation employed for this thus far? Would love to get
> opinions on (1) v/s (2).
>
> Thanks,
> Anirudh
>
>
>

Re: Integration testing and Scheduler Backends

[Anirudh Ramanathan]

I meant uncommon in Spark, i.e. in the other scheduler backends AFAIK.
(2) is what we use within the Kubernetes project for example, and is
preferred in general.


On Mon, Jan 8, 2018 at 10:45 PM, Felix Cheung 
wrote:

> How would (2) be uncommon elsewhere?
>
> On Mon, Jan 8, 2018 at 10:16 PM Anirudh Ramanathan 
> wrote:
>
>> This is with regard to the Kubernetes Scheduler Backend and scaling the
>> process to accept contributions. Given we're moving past upstreaming
>> changes from our fork, and into getting *new* patches, I wanted to start
>> this discussion sooner than later. This is more of a post-2.3 question -
>> not something we're looking to solve right away.
>>
>> While unit tests are handy, they're not nearly as good at giving us
>> confidence as a successful run of our integration tests against
>> single/multi-node k8s clusters. Currently, we have integration testing
>> setup at https://github.com/apache-spark-on-k8s/spark-integration and
>> it's running continuously against apache/spark:master in
>> pepperdata-jenkins
>>  (on
>> minikube) & k8s-testgrid
>>  
>> (in
>> GKE clusters). Now, the question is - how do we make integration-tests
>> part of the PR author's workflow?
>>
>> 1. Keep the integration tests in the separate repo and require that
>> contributors run them, add new tests prior to accepting their PRs as a
>> policy. Given minikube  is easy
>> to setup and can run on a single-node, it would certainly be possible.
>> Friction however, stems from contributors potentially having to modify the
>> integration test code hosted in that separate repository when
>> adding/changing functionality in the scheduler backend. Also, it's
>> certainly going to lead to at least brief inconsistencies between the two
>> repositories.
>>
>> 2. Alternatively, we check in the integration tests alongside the actual
>> scheduler backend code. This would work really well and is what we did in
>> our fork. It would have to be a separate package which would take certain
>> parameters (like cluster endpoint) and run integration test code against a
>> local or remote cluster. It would include least some code dealing with
>> accessing the cluster, reading results from K8s containers, test fixtures,
>> etc.
>>
>> I see value in adopting (2), given it's a clearer path for contributors
>> and lets us keep the two pieces consistent, but it seems uncommon
>> elsewhere. How do the other backends, i.e. YARN, Mesos and Standalone deal
>> with accepting patches and ensuring that they do not break existing
>> clusters? Is there automation employed for this thus far? Would love to get
>> opinions on (1) v/s (2).
>>
>> Thanks,
>> Anirudh
>>
>>
>>


-- 
Anirudh Ramanathan

Re: Integration testing and Scheduler Backends

[Timothy Chen]

2) will be ideal but given the velocity of main branch, what Mesos
ended up doing was simply having a separate repo since it will take
too long to merge back to main.

We ended up running it pre-release (or major PR merged) and not on
every PR, I will also comment on asking users to run it.

We did have conversations with Reynold about potentially have the
ability to run the CI on every [Mesos] tagged PR but we never got
there.

Tim

On Mon, Jan 8, 2018 at 10:16 PM, Anirudh Ramanathan
 wrote:
> This is with regard to the Kubernetes Scheduler Backend and scaling the
> process to accept contributions. Given we're moving past upstreaming changes
> from our fork, and into getting new patches, I wanted to start this
> discussion sooner than later. This is more of a post-2.3 question - not
> something we're looking to solve right away.
>
> While unit tests are handy, they're not nearly as good at giving us
> confidence as a successful run of our integration tests against
> single/multi-node k8s clusters. Currently, we have integration testing setup
> at https://github.com/apache-spark-on-k8s/spark-integration and it's running
> continuously against apache/spark:master in pepperdata-jenkins (on minikube)
> & k8s-testgrid (in GKE clusters). Now, the question is - how do we make
> integration-tests part of the PR author's workflow?
>
> 1. Keep the integration tests in the separate repo and require that
> contributors run them, add new tests prior to accepting their PRs as a
> policy. Given minikube is easy to setup and can run on a single-node, it
> would certainly be possible. Friction however, stems from contributors
> potentially having to modify the integration test code hosted in that
> separate repository when adding/changing functionality in the scheduler
> backend. Also, it's certainly going to lead to at least brief
> inconsistencies between the two repositories.
>
> 2. Alternatively, we check in the integration tests alongside the actual
> scheduler backend code. This would work really well and is what we did in
> our fork. It would have to be a separate package which would take certain
> parameters (like cluster endpoint) and run integration test code against a
> local or remote cluster. It would include least some code dealing with
> accessing the cluster, reading results from K8s containers, test fixtures,
> etc.
>
> I see value in adopting (2), given it's a clearer path for contributors and
> lets us keep the two pieces consistent, but it seems uncommon elsewhere. How
> do the other backends, i.e. YARN, Mesos and Standalone deal with accepting
> patches and ensuring that they do not break existing clusters? Is there
> automation employed for this thus far? Would love to get opinions on (1) v/s
> (2).
>
> Thanks,
> Anirudh
>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org