[jira] [Commented] (SPARK-26342) Support for NFS mount for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-26342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721997#comment-16721997 ] Yinan Li commented on SPARK-26342: -- Yes, that's true. Feel free to create a PR to add nfs and flex. >

[jira] [Resolved] (SPARK-26290) [K8s] Driver Pods no mounted volumes on submissions from older spark versions

[ https://issues.apache.org/jira/browse/SPARK-26290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li resolved SPARK-26290. -- Resolution: Not A Bug > [K8s] Driver Pods no mounted volumes on submissions from older spark versions

[jira] [Commented] (SPARK-26342) Support for NFS mount for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-26342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721894#comment-16721894 ] Yinan Li commented on SPARK-26342: -- So basically what you want is a generic way to mount arbitrary

[jira] [Commented] (SPARK-26344) Support for flexVolume mount for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-26344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721893#comment-16721893 ] Yinan Li commented on SPARK-26344: -- This is covered by SPARK-24434, which enables using a pod template

[jira] [Resolved] (SPARK-25515) Add a config property for disabling auto deletion of PODS for debugging.

[ https://issues.apache.org/jira/browse/SPARK-25515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li resolved SPARK-25515. -- Resolution: Fixed Fix Version/s: 3.0.0 > Add a config property for disabling auto deletion of

[jira] [Commented] (SPARK-25922) [K8] Spark Driver/Executor "spark-app-selector" label mismatch

[ https://issues.apache.org/jira/browse/SPARK-25922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677276#comment-16677276 ] Yinan Li commented on SPARK-25922: -- The application ID used to set the {{spark-app-selector}} label for

[jira] [Commented] (SPARK-25787) [K8S] Spark can't use data locality information

[ https://issues.apache.org/jira/browse/SPARK-25787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659593#comment-16659593 ] Yinan Li commented on SPARK-25787: -- Support for data locality on k8s has not been ported to the 

[jira] [Commented] (SPARK-25796) Enable external shuffle service for kubernetes mode.

[ https://issues.apache.org/jira/browse/SPARK-25796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659579#comment-16659579 ] Yinan Li commented on SPARK-25796: -- See https://issues.apache.org/jira/browse/SPARK-24432.  > Enable

[jira] [Updated] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24432: - Affects Version/s: 3.0.0 > Add support for dynamic resource allocation >

[jira] [Commented] (SPARK-25742) Is there a way to pass the Azure blob storage credentials to the spark for k8s init-container?

[ https://issues.apache.org/jira/browse/SPARK-25742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652064#comment-16652064 ] Yinan Li commented on SPARK-25742: -- The k8s secrets you add through the

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644198#comment-16644198 ] Yinan Li commented on SPARK-25682: -- Cool, thanks! > Docker images generated from dev build and from

[jira] [Commented] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644157#comment-16644157 ] Yinan Li commented on SPARK-25682: -- That looks like to me the only difference. {{bin}}, {{sbin}}, and

[jira] [Comment Edited] (SPARK-25500) Specify configmap and secrets in Spark driver and executor pods in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-25500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625404#comment-16625404 ] Yinan Li edited comment on SPARK-25500 at 9/24/18 5:51 AM: --- We don't plan to

[jira] [Commented] (SPARK-25500) Specify configmap and secrets in Spark driver and executor pods in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-25500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625404#comment-16625404 ] Yinan Li commented on SPARK-25500: -- We don't plan to add more configuration properties for pod

[jira] [Resolved] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li resolved SPARK-23200. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22392

[jira] [Resolved] (SPARK-25291) Flakiness of tests in terms of executor memory (SecretsTestSuite)

[ https://issues.apache.org/jira/browse/SPARK-25291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li resolved SPARK-25291. -- Resolution: Fixed Fix Version/s: 2.4.0 > Flakiness of tests in terms of executor memory

[jira] [Resolved] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li resolved SPARK-25295. -- Resolution: Fixed Fix Version/s: 2.4.0 > Pod names conflicts in client mode, if previous

[jira] [Issue Comment Deleted] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-25295: - Comment: was deleted (was: We made it clear in the documentation of the Kubernetes mode at

[jira] [Commented] (SPARK-25282) Fix support for spark-shell with K8s

[ https://issues.apache.org/jira/browse/SPARK-25282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599310#comment-16599310 ] Yinan Li commented on SPARK-25282: -- I'm not sure this is a bug and how this should be enforced

[jira] [Commented] (SPARK-25295) Pod names conflicts in client mode, if previous submission was not a clean shutdown.

[ https://issues.apache.org/jira/browse/SPARK-25295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599308#comment-16599308 ] Yinan Li commented on SPARK-25295: -- We made it clear in the documentation of the Kubernetes mode at

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599304#comment-16599304 ] Yinan Li commented on SPARK-24434: -- [~skonto] we can understand your feeling and frustration on this,

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594038#comment-16594038 ] Yinan Li commented on SPARK-24434: -- It seemed I couldn't change the assignee. > Support user-specified

[jira] [Commented] (SPARK-25162) Kubernetes 'in-cluster' client mode and value of spark.driver.host

[ https://issues.apache.org/jira/browse/SPARK-25162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589286#comment-16589286 ] Yinan Li commented on SPARK-25162: -- > Where the driver is running _outside-cluster client_ mode,  would

[jira] [Commented] (SPARK-25194) Kubernetes - Define cpu and memory limit to init container

[ https://issues.apache.org/jira/browse/SPARK-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589283#comment-16589283 ] Yinan Li commented on SPARK-25194: -- The upcoming Spark 2.4 gets rid of the init-container and switch to

[jira] [Commented] (SPARK-25162) Kubernetes 'in-cluster' client mode and value of spark.driver.host

[ https://issues.apache.org/jira/browse/SPARK-25162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588014#comment-16588014 ] Yinan Li commented on SPARK-25162: -- We actually moved away from using the IP address of the driver pod

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586314#comment-16586314 ] Yinan Li commented on SPARK-24434: -- [~skonto] I will make sure the assignee gets properly set for

[jira] [Commented] (SPARK-25066) Provide Spark R image for deploying Spark on kubernetes.

[ https://issues.apache.org/jira/browse/SPARK-25066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576555#comment-16576555 ] Yinan Li commented on SPARK-25066: -- R support is still being worked on and will likely go into 2.4. Is

[jira] [Commented] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560436#comment-16560436 ] Yinan Li commented on SPARK-24724: -- Sorry haven't got a chance to look into this. What pieces of info

[jira] [Commented] (SPARK-24894) Invalid DNS name due to hostname truncation

[ https://issues.apache.org/jira/browse/SPARK-24894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554602#comment-16554602 ] Yinan Li commented on SPARK-24894: -- [~mcheah]. We need to make sure the truncation leads to a valid

[jira] [Updated] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24724: - Component/s: Kubernetes > Discuss necessary info and access in barrier mode + Kubernetes >

[jira] [Comment Edited] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542109#comment-16542109 ] Yinan Li edited comment on SPARK-24793 at 7/12/18 7:11 PM: --- Oh, yeah, {{kill}} 

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542109#comment-16542109 ] Yinan Li commented on SPARK-24793: -- Oh, yeah, {{kill}} and {{status}} are existing options of

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542003#comment-16542003 ] Yinan Li commented on SPARK-24793: -- Good points, Erik. I think

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540252#comment-16540252 ] Yinan Li commented on SPARK-24432: -- No one is working on this right now, but I think foxish planned to

[jira] [Commented] (SPARK-24765) Add custom Kubernetes scheduler config parameter to spark-submit

[ https://issues.apache.org/jira/browse/SPARK-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538999#comment-16538999 ] Yinan Li commented on SPARK-24765: -- Check out https://issues.apache.org/jira/browse/SPARK-24434 and 

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513064#comment-16513064 ] Yinan Li commented on SPARK-24434: -- [~skonto] Thanks! Will take a look at the design doc once I'm back

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499220#comment-16499220 ] Yinan Li commented on SPARK-24434: -- [~skonto] Thanks for the detailed thoughts! I agree with you that

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498307#comment-16498307 ] Yinan Li edited comment on SPARK-24434 at 6/1/18 5:39 PM: -- The pod template is

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498307#comment-16498307 ] Yinan Li edited comment on SPARK-24434 at 6/1/18 5:38 PM: -- The pod template is

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498307#comment-16498307 ] Yinan Li commented on SPARK-24434: -- The pod template is basically a pod specification and can contain

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496967#comment-16496967 ] Yinan Li commented on SPARK-24434: -- [~foxish] that sounds like the approach to go.  > Support

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495637#comment-16495637 ] Yinan Li commented on SPARK-24434: -- [~eje] That's a good question. I think we need to compare both and

[jira] [Created] (SPARK-24434) Support user-specified driver and executor pod templates

Yinan Li created SPARK-24434: Summary: Support user-specified driver and executor pod templates Key: SPARK-24434 URL: https://issues.apache.org/jira/browse/SPARK-24434 Project: Spark Issue Type:

[jira] [Created] (SPARK-24433) Add Spark R support

Yinan Li created SPARK-24433: Summary: Add Spark R support Key: SPARK-24433 URL: https://issues.apache.org/jira/browse/SPARK-24433 Project: Spark Issue Type: New Feature Components:

[jira] [Created] (SPARK-24432) Support for dynamic resource allocation

Yinan Li created SPARK-24432: Summary: Support for dynamic resource allocation Key: SPARK-24432 URL: https://issues.apache.org/jira/browse/SPARK-24432 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24432: - Summary: Add support for dynamic resource allocation (was: Support for dynamic resource allocation) >

[jira] [Commented] (SPARK-24122) Allow automatic driver restarts on K8s

[ https://issues.apache.org/jira/browse/SPARK-24122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491338#comment-16491338 ] Yinan Li commented on SPARK-24122: -- The operator does cover automatic restart of an application with a

[jira] [Commented] (SPARK-24091) Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

[ https://issues.apache.org/jira/browse/SPARK-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491279#comment-16491279 ] Yinan Li commented on SPARK-24091: -- Thanks [~tmckay]! I think the first approach is a good way of

[jira] [Commented] (SPARK-24383) spark on k8s: "driver-svc" are not getting deleted

[ https://issues.apache.org/jira/browse/SPARK-24383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491106#comment-16491106 ] Yinan Li commented on SPARK-24383: -- OK, then garbage collection should kick in and delete the service

[jira] [Commented] (SPARK-24383) spark on k8s: "driver-svc" are not getting deleted

[ https://issues.apache.org/jira/browse/SPARK-24383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489942#comment-16489942 ] Yinan Li commented on SPARK-24383: -- You can use {{kubectl get service -o=yaml}} to get a

[jira] [Commented] (SPARK-24383) spark on k8s: "driver-svc" are not getting deleted

[ https://issues.apache.org/jira/browse/SPARK-24383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489776#comment-16489776 ] Yinan Li commented on SPARK-24383: -- Can you double check if the services have an {{OwnerReference}}

[jira] [Commented] (SPARK-24383) spark on k8s: "driver-svc" are not getting deleted

[ https://issues.apache.org/jira/browse/SPARK-24383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489601#comment-16489601 ] Yinan Li commented on SPARK-24383: -- The Kubernetes specific submission client adds an {{OwnerReference}} 

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477825#comment-16477825 ] Yinan Li commented on SPARK-24248: -- Re-sync is not a fallback nor a replacement, but a complement to the

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472574#comment-16472574 ] Yinan Li commented on SPARK-24232: -- As long as we document it clearly what is for, I think it's OK,

[jira] [Comment Edited] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472561#comment-16472561 ] Yinan Li edited comment on SPARK-24232 at 5/11/18 7:55 PM: --- We should keep the

[jira] [Commented] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472561#comment-16472561 ] Yinan Li commented on SPARK-24232: -- We should keep the current semantics of

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471479#comment-16471479 ] Yinan Li commented on SPARK-24248: -- I think it's both more robust and easier to implement with a

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471288#comment-16471288 ] Yinan Li commented on SPARK-24248: -- Just realized one thing: solely replying on the watcher poses risks

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471259#comment-16471259 ] Yinan Li commented on SPARK-24248: -- Actually even if the fabric8 client does not support caching, we can

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471244#comment-16471244 ] Yinan Li commented on SPARK-24248: -- It's potentially possible to get rid of the in-memory state in favor

[jira] [Updated] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24137: - Fix Version/s: (was: 2.3.1) > [K8s] Mount temporary directories in emptydir volumes >

[jira] [Updated] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24137: - Fix Version/s: 2.3.1 > [K8s] Mount temporary directories in emptydir volumes >

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460066#comment-16460066 ] Yinan Li edited comment on SPARK-24135 at 5/1/18 7:53 PM: -- I agree that we

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460066#comment-16460066 ] Yinan Li commented on SPARK-24135: -- I agree that we should add detection for initialization errors. But

[jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459900#comment-16459900 ] Yinan Li commented on SPARK-24137: -- Yeah, {{LocalDirectoryMountConfigurationStep}} was missed in the

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459892#comment-16459892 ] Yinan Li commented on SPARK-24135: -- I think it's fine detecting and deleting the executor pods that

[jira] [Updated] (SPARK-24091) Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

[ https://issues.apache.org/jira/browse/SPARK-24091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24091: - Affects Version/s: (was: 2.3.0) 2.4.0 > Internally used ConfigMap prevents

[jira] [Created] (SPARK-24091) Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files

Yinan Li created SPARK-24091: Summary: Internally used ConfigMap prevents use of user-specified ConfigMaps carrying Spark configs files Key: SPARK-24091 URL: https://issues.apache.org/jira/browse/SPARK-24091

[jira] [Resolved] (SPARK-23638) Spark on k8s: spark.kubernetes.initContainer.image has no effect

[ https://issues.apache.org/jira/browse/SPARK-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li resolved SPARK-23638. -- Resolution: Not A Problem > Spark on k8s: spark.kubernetes.initContainer.image has no effect >

[jira] [Commented] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444899#comment-16444899 ] Yinan Li commented on SPARK-24028: -- 2.3.0 does create a configmap for the init-container if one is used.

[jira] [Comment Edited] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444890#comment-16444890 ] Yinan Li edited comment on SPARK-24028 at 4/19/18 10:14 PM: I run a 1.9.6

[jira] [Commented] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444890#comment-16444890 ] Yinan Li commented on SPARK-24028: -- I run a 1.9.6 cluster. No, I was using the 2.3.0 release. > [K8s]

[jira] [Commented] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444856#comment-16444856 ] Yinan Li commented on SPARK-24028: -- I am also running a 1.9 cluster on GKE and I have never run into the

[jira] [Commented] (SPARK-23638) Spark on k8s: spark.kubernetes.initContainer.image has no effect

[ https://issues.apache.org/jira/browse/SPARK-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440067#comment-16440067 ] Yinan Li commented on SPARK-23638: -- Can this be closed? > Spark on k8s:

[jira] [Commented] (SPARK-23638) Spark on k8s: spark.kubernetes.initContainer.image has no effect

[ https://issues.apache.org/jira/browse/SPARK-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16402070#comment-16402070 ] Yinan Li commented on SPARK-23638: -- The Kubernetes-specific submission client will only add an

[jira] [Updated] (SPARK-23571) Delete auxiliary Kubernetes resources upon application completion

[ https://issues.apache.org/jira/browse/SPARK-23571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-23571: - Affects Version/s: 2.3.1 > Delete auxiliary Kubernetes resources upon application completion >

[jira] [Created] (SPARK-23571) Delete auxiliary Kubernetes resources upon application completion

Yinan Li created SPARK-23571: Summary: Delete auxiliary Kubernetes resources upon application completion Key: SPARK-23571 URL: https://issues.apache.org/jira/browse/SPARK-23571 Project: Spark

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374757#comment-16374757 ] Yinan Li edited comment on SPARK-23485 at 2/23/18 6:22 PM: --- It's not that I'm 

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374757#comment-16374757 ] Yinan Li commented on SPARK-23485: -- It's not that I'm too confident on the capability of Kubernetes to

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374708#comment-16374708 ] Yinan Li commented on SPARK-23485: -- In the Yarn case, yes, it's possible that a node is missing a jar

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373620#comment-16373620 ] Yinan Li commented on SPARK-23485: -- The Kubernetes scheduler backend simply creates executor pods

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373544#comment-16373544 ] Yinan Li commented on SPARK-23485: -- I'm not sure if node blacklisting applies to Kubernetes. In the

[jira] [Comment Edited] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357500#comment-16357500 ] Yinan Li edited comment on SPARK-23285 at 2/8/18 8:22 PM: -- Given the complexity

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357500#comment-16357500 ] Yinan Li commented on SPARK-23285: -- Given the complexity and significant impact of the changes proposed

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347484#comment-16347484 ] Yinan Li commented on SPARK-23285: -- Another option is to bypass that check for Kubernetes mode. This

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347267#comment-16347267 ] Yinan Li commented on SPARK-23285: -- FYI: we did this in our fork: 

[jira] [Commented] (SPARK-23257) Implement Kerberos Support in Kubernetes resource manager

[ https://issues.apache.org/jira/browse/SPARK-23257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345488#comment-16345488 ] Yinan Li commented on SPARK-23257: -- [~RJKeevil] AFAIK, no one is working on upstreaming this yet.

[jira] [Created] (SPARK-23153) Support application dependencies in submission client's local file system

Yinan Li created SPARK-23153: Summary: Support application dependencies in submission client's local file system Key: SPARK-23153 URL: https://issues.apache.org/jira/browse/SPARK-23153 Project: Spark

[jira] [Commented] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331132#comment-16331132 ] Yinan Li commented on SPARK-22962: -- I agree that before we upstream the staging server, we should fail

[jira] [Commented] (SPARK-23137) spark.kubernetes.executor.podNamePrefix is ignored

[ https://issues.apache.org/jira/browse/SPARK-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329713#comment-16329713 ] Yinan Li commented on SPARK-23137: -- It's actually marked as an \{{internal}} config property. So the fix

[jira] [Created] (SPARK-22998) Value for SPARK_MOUNTED_CLASSPATH in executor pods is not set

Yinan Li created SPARK-22998: Summary: Value for SPARK_MOUNTED_CLASSPATH in executor pods is not set Key: SPARK-22998 URL: https://issues.apache.org/jira/browse/SPARK-22998 Project: Spark Issue

[jira] [Created] (SPARK-22953) Duplicated secret volumes in Spark pods when init-containers are used

Yinan Li created SPARK-22953: Summary: Duplicated secret volumes in Spark pods when init-containers are used Key: SPARK-22953 URL: https://issues.apache.org/jira/browse/SPARK-22953 Project: Spark

[jira] [Created] (SPARK-22839) Refactor Kubernetes code for configuring driver/executor pods to use consistent and cleaner abstraction

Yinan Li created SPARK-22839: Summary: Refactor Kubernetes code for configuring driver/executor pods to use consistent and cleaner abstraction Key: SPARK-22839 URL: https://issues.apache.org/jira/browse/SPARK-22839

[jira] [Commented] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289907#comment-16289907 ] Yinan Li commented on SPARK-22778: -- Just verified that the fix worked. I'm gonna send a PR soon. >

[jira] [Commented] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289876#comment-16289876 ] Yinan Li commented on SPARK-22778: -- Ah, yes, the PR missed that. OK, I'm gonna give that a try and

[jira] [Comment Edited] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289822#comment-16289822 ] Yinan Li edited comment on SPARK-22778 at 12/13/17 8:24 PM: Just some

[jira] [Commented] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289822#comment-16289822 ] Yinan Li commented on SPARK-22778: -- Just some background on this. The validation and parsing of k8s

[jira] [Updated] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-18278: - Component/s: Kubernetes > SPIP: Support native submission of spark jobs to a kubernetes cluster >

[jira] [Commented] (SPARK-22757) Init-container in the driver/executor pods for downloading remote dependencies

[ https://issues.apache.org/jira/browse/SPARK-22757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289378#comment-16289378 ] Yinan Li commented on SPARK-22757: -- Yes, this is also targeting 2.3. > Init-container in the

[jira] [Updated] (SPARK-22757) Init-container in the driver/executor pods for downloading remote dependencies

[ https://issues.apache.org/jira/browse/SPARK-22757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-22757: - Component/s: Kubernetes > Init-container in the driver/executor pods for downloading remote dependencies

  1   2   >