[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
Kubernetes integration test status failure
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2316/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
Kubernetes integration test starting
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2316/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2361/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
**[Test build #89418 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89418/testReport)**
 for PR 21032 at commit 
[`86afec3`](https://github.com/apache/spark/commit/86afec32ed60735bef78d93f52de12e8b006).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89418/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
**[Test build #89418 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89418/testReport)**
 for PR 21032 at commit 
[`86afec3`](https://github.com/apache/spark/commit/86afec32ed60735bef78d93f52de12e8b006).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[madanadit]

Github user madanadit commented on the issue:

https://github.com/apache/spark/pull/21032
  
@liyinan926 @foxish I've started a doc 
[here](https://docs.google.com/document/d/15-mk7UnOYNTXoF6EKaVlelWYc9DTrTXrYoodwDuAwY4/edit?usp=sharing).
 Feel free to edit and circulate to interested members of the community.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/21032
  
@madanadit @liyinan926, can we do a short doc with all the types of volumes 
and config options and run that through the rest of the community?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[madanadit]

Github user madanadit commented on the issue:

https://github.com/apache/spark/pull/21032
  
@liyinan926 Sounds good


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[liyinan926]

Github user liyinan926 commented on the issue:

https://github.com/apache/spark/pull/21032
  
@madanadit personally I like the first proposal better. The only change I'm 
proposing is split `spark.kubernetes.executor.volumes.[type].[name].mount` into 
two keys: `spark.kubernetes.executor.volumes.[type].[name].mount.path` and 
`spark.kubernetes.executor.volumes.[type].[name].mount.readOnly`, which is 
`false` by default.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[madanadit]

Github user madanadit commented on the issue:

https://github.com/apache/spark/pull/21032
  
@foxish @liyinan926 Thanks for your comments. To make the configuration 
more general, here is an initial proposal. 

Context:
- volumeMount requires `[name, mountPath, readWriteMode]`
- volume requires `[name, type, [type_specific_options…]]`
for `type=hostPath`, `type_specific_options=[path, 
type=[Directory,File,…]]`

Proposed format :
```properties

spark.kubernetes.executor.volumes.[type].[name].mount=mountPath:readWriteMode

spark.kubernetes.executor.volumes.[type].[name].options.[type_specific_option1]=[value]
```
where, the format for value can be chosen based on the volume type/option. 
Nested options can be easily supported using this format. For instance, for 
hostPath volumes the config would look like:

```properties
spark.kubernetes.executor.volumes.hostPath.hostPath-1.mount=/opt/domain:ro

spark.kubernetes.executor.volumes.hostPath.hostPath-1.options.path=/tmp/domain
spark.kubernetes.executor.volumes.hostPath.hostPath-1.options.type=Directory
```
Alternatively, the volume type can be a top level property like 
`spark.kubernetes.executor.volumes.[name]=[type]`. Thoughts?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/21032
  
Yes, that is what I was indicating as well. We should have a design for all
different volume types. Extending this PR as the design evolves is a good
way to go about that.

On Thu, Apr 12, 2018, 11:37 AM Yinan Li  wrote:

> I think we should have a more general solution so it works for other types
> of volumes. Particularly, we should have a design discussion on the format
> of values of such a property so it works for any type of volumes. I 
suggest
> we hold on on this PR until we setup a general solution. @foxish
>  @felixcheung 
> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[liyinan926]

Github user liyinan926 commented on the issue:

https://github.com/apache/spark/pull/21032
  
I think we should have a more general solution so it works for other types 
of volumes. Particularly, we should have a design discussion on the format of 
values of such a property so it works for any type of volumes. I suggest we 
hold on on this PR until we setup a general solution. @foxish @felixcheung .


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
Kubernetes integration test status success
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2240/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
Kubernetes integration test starting
URL: 
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2240/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2284/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
**[Test build #89291 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89291/testReport)**
 for PR 21032 at commit 
[`c72cbc1`](https://github.com/apache/spark/commit/c72cbc1bd8a5043f5dbc55fa7ba0631c694a9317).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89291/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21032
  
**[Test build #89291 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89291/testReport)**
 for PR 21032 at commit 
[`c72cbc1`](https://github.com/apache/spark/commit/c72cbc1bd8a5043f5dbc55fa7ba0631c694a9317).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/21032
  
Jenkins, ok to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[madanadit]

Github user madanadit commented on the issue:

https://github.com/apache/spark/pull/21032
  
@felixcheung, @foxish can you take a look? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21032: [SPARK-23529][K8s] Support mounting hostPath volumes for...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21032
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org