[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542135#comment-16542135 ] Anirudh Ramanathan commented on SPARK-24793: Great! I'll take a stab at a PR in a few days.

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541990#comment-16541990 ] Anirudh Ramanathan commented on SPARK-24793: Good points Erik. These options are not without

[jira] [Updated] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24793: --- Description: Support controlling the lifecycle of Spark Application through

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541297#comment-16541297 ] Anirudh Ramanathan commented on SPARK-24793: This came up in our spark summit BoF session as

[jira] [Updated] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24793: --- Description: Support controlling the lifecycle of Spark Application through

[jira] [Created] (SPARK-24793) Make spark-submit more useful with k8s

Anirudh Ramanathan created SPARK-24793: -- Summary: Make spark-submit more useful with k8s Key: SPARK-24793 URL: https://issues.apache.org/jira/browse/SPARK-24793 Project: Spark Issue

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541272#comment-16541272 ] Anirudh Ramanathan commented on SPARK-24434: Thanks for kicking this design off Stavros.

[jira] [Commented] (SPARK-24432) Add support for dynamic resource allocation

[ https://issues.apache.org/jira/browse/SPARK-24432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541239#comment-16541239 ] Anirudh Ramanathan commented on SPARK-24432: Hi Mark, we did it once before in our fork in a

[jira] [Resolved] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23885. Resolution: Not A Bug > trying to spark submit 2.3.0 on minikube >

[jira] [Commented] (SPARK-24428) Remove unused code and fix any related doc in K8s module

[ https://issues.apache.org/jira/browse/SPARK-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530391#comment-16530391 ] Anirudh Ramanathan commented on SPARK-24428: Resolved by

[jira] [Resolved] (SPARK-24428) Remove unused code and fix any related doc in K8s module

[ https://issues.apache.org/jira/browse/SPARK-24428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24428. Resolution: Fixed Fix Version/s: 2.4.0 > Remove unused code and fix any

[jira] [Resolved] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24547. Resolution: Fixed Fix Version/s: 2.4.0 > Spark on K8s docker-image-tool.sh

[jira] [Commented] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518738#comment-16518738 ] Anirudh Ramanathan commented on SPARK-24547: Resolved by

[jira] [Assigned] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24547: -- Assignee: (was: Anirudh Ramanathan) > Spark on K8s docker-image-tool.sh

[jira] [Assigned] (SPARK-24547) Spark on K8s docker-image-tool.sh improvements

[ https://issues.apache.org/jira/browse/SPARK-24547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24547: -- Assignee: Anirudh Ramanathan > Spark on K8s docker-image-tool.sh

[jira] [Updated] (SPARK-24600) Improve support for building different types of images in dockerfile

[ https://issues.apache.org/jira/browse/SPARK-24600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24600: --- Description: Our docker images currently build and push docker images for pyspark

[jira] [Created] (SPARK-24600) Improve support for building subset of images in dockerfile

Anirudh Ramanathan created SPARK-24600: -- Summary: Improve support for building subset of images in dockerfile Key: SPARK-24600 URL: https://issues.apache.org/jira/browse/SPARK-24600 Project:

[jira] [Updated] (SPARK-24600) Improve support for building different types of images in dockerfile

[ https://issues.apache.org/jira/browse/SPARK-24600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24600: --- Summary: Improve support for building different types of images in dockerfile

[jira] [Resolved] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24232. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24232: -- Assignee: Stavros Kontopoulos > Allow referring to kubernetes secrets as env

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497014#comment-16497014 ] Anirudh Ramanathan commented on SPARK-24434: Open to suggestions on what could be intuitive

[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497009#comment-16497009 ] Anirudh Ramanathan edited comment on SPARK-24434 at 5/31/18 6:54 PM: -

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497009#comment-16497009 ] Anirudh Ramanathan commented on SPARK-24434: Good point. I was basing my suggestion of JSON

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496962#comment-16496962 ] Anirudh Ramanathan commented on SPARK-24434: The way several custom APIs have done this

[jira] [Comment Edited] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474774#comment-16474774 ] Anirudh Ramanathan edited comment on SPARK-24248 at 5/14/18 8:44 PM: -

[jira] [Commented] (SPARK-24248) [K8S] Use the Kubernetes cluster as the backing store for the state of pods

[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474774#comment-16474774 ] Anirudh Ramanathan commented on SPARK-24248: Somewhat related -

[jira] [Commented] (SPARK-24249) Spark on kubernetes, pods crashes with spark sql job.

[ https://issues.apache.org/jira/browse/SPARK-24249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474772#comment-16474772 ] Anirudh Ramanathan commented on SPARK-24249: cc/ [~mccheah] > Spark on kubernetes, pods

[jira] [Commented] (SPARK-24266) Spark client terminates while driver is still running

[ https://issues.apache.org/jira/browse/SPARK-24266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474770#comment-16474770 ] Anirudh Ramanathan commented on SPARK-24266: Thanks for filing this issue and digging in. It

[jira] [Assigned] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-24137: -- Assignee: Matt Cheah > [K8s] Mount temporary directories in emptydir volumes

[jira] [Resolved] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-24137. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21238

[jira] [Updated] (SPARK-24179) History Server for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24179: --- Issue Type: New Feature (was: Task) > History Server for Kubernetes >

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462145#comment-16462145 ] Anirudh Ramanathan commented on SPARK-24135: cc/ [~mridulm80] [~irashid] for thoughts on

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462139#comment-16462139 ] Anirudh Ramanathan edited comment on SPARK-24135 at 5/3/18 9:01 AM:

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462139#comment-16462139 ] Anirudh Ramanathan edited comment on SPARK-24135 at 5/3/18 8:58 AM:

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462139#comment-16462139 ] Anirudh Ramanathan commented on SPARK-24135: It is increasingly common for people to write

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460559#comment-16460559 ] Anirudh Ramanathan commented on SPARK-24135: +1 to detecting all pod error states and doing

[jira] [Commented] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460554#comment-16460554 ] Anirudh Ramanathan commented on SPARK-24137: SGTM. Let's try and share as much code as

[jira] [Commented] (SPARK-24105) Spark 2.3.0 on kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454847#comment-16454847 ] Anirudh Ramanathan commented on SPARK-24105: > To avoid this deadlock, its required to

[jira] [Created] (SPARK-24090) Kubernetes Backend Hotlist for Spark 2.4

Anirudh Ramanathan created SPARK-24090: -- Summary: Kubernetes Backend Hotlist for Spark 2.4 Key: SPARK-24090 URL: https://issues.apache.org/jira/browse/SPARK-24090 Project: Spark Issue

[jira] [Updated] (SPARK-24055) Add e2e test for using kubectl proxy for submitting spark jobs

[ https://issues.apache.org/jira/browse/SPARK-24055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-24055: --- Summary: Add e2e test for using kubectl proxy for submitting spark jobs (was: Add

[jira] [Created] (SPARK-24055) Add e2e test for using kubectl proxy for submission

Anirudh Ramanathan created SPARK-24055: -- Summary: Add e2e test for using kubectl proxy for submission Key: SPARK-24055 URL: https://issues.apache.org/jira/browse/SPARK-24055 Project: Spark

[jira] [Commented] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444902#comment-16444902 ] Anirudh Ramanathan commented on SPARK-24028: My suspicion here is that this has to do with

[jira] [Comment Edited] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444902#comment-16444902 ] Anirudh Ramanathan edited comment on SPARK-24028 at 4/19/18 10:22 PM:

[jira] [Comment Edited] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444881#comment-16444881 ] Anirudh Ramanathan edited comment on SPARK-24028 at 4/19/18 10:09 PM:

[jira] [Commented] (SPARK-24028) [K8s] Creating secrets and config maps before creating the driver pod has unpredictable behavior

[ https://issues.apache.org/jira/browse/SPARK-24028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444881#comment-16444881 ] Anirudh Ramanathan commented on SPARK-24028: This is curious. Is this a recent change? You

[jira] [Resolved] (SPARK-22839) Refactor Kubernetes code for configuring driver/executor pods to use consistent and cleaner abstraction

[ https://issues.apache.org/jira/browse/SPARK-22839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-22839. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 20910

[jira] [Assigned] (SPARK-22839) Refactor Kubernetes code for configuring driver/executor pods to use consistent and cleaner abstraction

[ https://issues.apache.org/jira/browse/SPARK-22839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-22839: -- Assignee: Matt Cheah > Refactor Kubernetes code for configuring

[jira] [Comment Edited] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436094#comment-16436094 ] Anirudh Ramanathan edited comment on SPARK-23885 at 4/12/18 6:21 PM: -

[jira] [Commented] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436094#comment-16436094 ] Anirudh Ramanathan commented on SPARK-23885: Please post questions to the spark-user mailing

[jira] [Assigned] (SPARK-23885) trying to spark submit 2.3.0 on minikube

[ https://issues.apache.org/jira/browse/SPARK-23885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23885: -- Assignee: Anirudh Ramanathan > trying to spark submit 2.3.0 on minikube >

[jira] [Commented] (SPARK-23891) Debian based Dockerfile

[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436092#comment-16436092 ] Anirudh Ramanathan commented on SPARK-23891: [~eje] has done a lot of research on these

[jira] [Resolved] (SPARK-23668) Support for imagePullSecrets k8s option

[ https://issues.apache.org/jira/browse/SPARK-23668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23668. Resolution: Fixed > Support for imagePullSecrets k8s option >

[jira] [Assigned] (SPARK-23668) Support for imagePullSecrets k8s option

[ https://issues.apache.org/jira/browse/SPARK-23668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23668: -- Assignee: Andrew Korzhuev > Support for imagePullSecrets k8s option >

[jira] [Resolved] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23285. Resolution: Fixed > Allow spark.executor.cores to be fractional >

[jira] [Assigned] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23285: -- Assignee: Yinan Li > Allow spark.executor.cores to be fractional >

[jira] [Updated] (SPARK-22865) Publish Official Apache Spark Docker images

[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22865: --- Priority: Major (was: Minor) > Publish Official Apache Spark Docker images >

[jira] [Commented] (SPARK-22865) Publish Official Apache Spark Docker images

[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422992#comment-16422992 ] Anirudh Ramanathan commented on SPARK-22865: [~eje] was working on getting the licensing

[jira] [Commented] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422987#comment-16422987 ] Anirudh Ramanathan commented on SPARK-23680: [~felixcheung] helped me set up the right

[jira] [Comment Edited] (SPARK-23680) entrypoint.sh does not accept arbitrary UIDs, returning as an error

[ https://issues.apache.org/jira/browse/SPARK-23680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422987#comment-16422987 ] Anirudh Ramanathan edited comment on SPARK-23680 at 4/2/18 7:04 PM:

[jira] [Commented] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397432#comment-16397432 ] Anirudh Ramanathan commented on SPARK-23200: Looks like the change was reverted.

[jira] [Resolved] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-18278. Resolution: Fixed Closing this. Sub-issues have been created for remaining items

[jira] [Updated] (SPARK-22865) Publish Official Apache Spark Docker images

[ https://issues.apache.org/jira/browse/SPARK-22865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22865: --- Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-18278) >

[jira] [Updated] (SPARK-23067) Allow for easier debugging of the docker container

[ https://issues.apache.org/jira/browse/SPARK-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23067: --- Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-18278) > Allow for

[jira] [Updated] (SPARK-23010) Add integration testing for Kubernetes backend into the apache/spark repository

[ https://issues.apache.org/jira/browse/SPARK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23010: --- Issue Type: New Feature (was: Sub-task) Parent: (was: SPARK-18278) >

[jira] [Resolved] (SPARK-23324) Announce new Kubernetes back-end for 2.3 release notes

[ https://issues.apache.org/jira/browse/SPARK-23324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23324. Resolution: Fixed > Announce new Kubernetes back-end for 2.3 release notes >

[jira] [Assigned] (SPARK-23324) Announce new Kubernetes back-end for 2.3 release notes

[ https://issues.apache.org/jira/browse/SPARK-23324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23324: -- Assignee: Erik Erlandson > Announce new Kubernetes back-end for 2.3 release

[jira] [Commented] (SPARK-23529) Specify hostpath volume and mount the volume in Spark driver and executor pods in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-23529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397421#comment-16397421 ] Anirudh Ramanathan commented on SPARK-23529: Can this wait till we have local PVs? Mounting

[jira] [Commented] (SPARK-23618) docker-image-tool.sh Fails While Building Image

[ https://issues.apache.org/jira/browse/SPARK-23618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397417#comment-16397417 ] Anirudh Ramanathan commented on SPARK-23618: Thanks Felix. The mechanism appears to work, but

[jira] [Assigned] (SPARK-23618) docker-image-tool.sh Fails While Building Image

[ https://issues.apache.org/jira/browse/SPARK-23618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23618: -- Assignee: (was: Anirudh Ramanathan) > docker-image-tool.sh Fails While

[jira] [Assigned] (SPARK-23618) docker-image-tool.sh Fails While Building Image

[ https://issues.apache.org/jira/browse/SPARK-23618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23618: -- Assignee: Anirudh Ramanathan > docker-image-tool.sh Fails While Building

[jira] [Assigned] (SPARK-23136) Mark packages as experimental

[ https://issues.apache.org/jira/browse/SPARK-23136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23136: -- Assignee: Anirudh Ramanathan > Mark packages as experimental >

[jira] [Assigned] (SPARK-23016) Spark UI access and documentation

[ https://issues.apache.org/jira/browse/SPARK-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23016: -- Assignee: Anirudh Ramanathan > Spark UI access and documentation >

[jira] [Assigned] (SPARK-22758) New Spark Jira component for Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-22758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-22758: -- Assignee: Marcelo Vanzin > New Spark Jira component for Kubernetes >

[jira] [Assigned] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-22647: -- Assignee: Yinan Li > Docker files for image creation >

[jira] [Assigned] (SPARK-22645) Add Scheduler Backend with static allocation of executors

[ https://issues.apache.org/jira/browse/SPARK-22645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-22645: -- Assignee: Anirudh Ramanathan > Add Scheduler Backend with static allocation

[jira] [Commented] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397403#comment-16397403 ] Anirudh Ramanathan commented on SPARK-18278: I think we should close this JIRA out and move

[jira] [Assigned] (SPARK-23083) Adding Kubernetes as an option to https://spark.apache.org/

[ https://issues.apache.org/jira/browse/SPARK-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-23083: -- Assignee: Anirudh Ramanathan > Adding Kubernetes as an option to

[jira] [Assigned] (SPARK-18278) SPIP: Support native submission of spark jobs to a kubernetes cluster

[ https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan reassigned SPARK-18278: -- Assignee: Anirudh Ramanathan > SPIP: Support native submission of spark jobs

[jira] [Commented] (SPARK-23618) docker-image-tool.sh Fails While Building Image

[ https://issues.apache.org/jira/browse/SPARK-23618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395686#comment-16395686 ] Anirudh Ramanathan commented on SPARK-23618: [~felixcheung], just merged this PR. But I'm

[jira] [Resolved] (SPARK-23618) docker-image-tool.sh Fails While Building Image

[ https://issues.apache.org/jira/browse/SPARK-23618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23618. Resolution: Fixed > docker-image-tool.sh Fails While Building Image >

[jira] [Resolved] (SPARK-23083) Adding Kubernetes as an option to https://spark.apache.org/

[ https://issues.apache.org/jira/browse/SPARK-23083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23083. Resolution: Fixed This has been merged, closing. > Adding Kubernetes as an option

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374884#comment-16374884 ] Anirudh Ramanathan edited comment on SPARK-23485 at 2/23/18 7:36 PM: -

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374884#comment-16374884 ] Anirudh Ramanathan commented on SPARK-23485: Stavros - we [do currently

[jira] [Comment Edited] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374745#comment-16374745 ] Anirudh Ramanathan edited comment on SPARK-23485 at 2/23/18 6:00 PM: -

[jira] [Commented] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374745#comment-16374745 ] Anirudh Ramanathan commented on SPARK-23485: While mostly I think that K8s would be better

[jira] [Updated] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23285: --- Description: There is a strong check for an integral number of cores per executor in

[jira] [Updated] (SPARK-23285) Allow spark.executor.cores to be fractional

[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23285: --- Description: There is a strong check for an integral number of cores per executor in

[jira] [Created] (SPARK-23285) Allow spark.executor.cores to be fractional

Anirudh Ramanathan created SPARK-23285: -- Summary: Allow spark.executor.cores to be fractional Key: SPARK-23285 URL: https://issues.apache.org/jira/browse/SPARK-23285 Project: Spark

[jira] [Created] (SPARK-23200) Reset configuration when restarting from checkpoints

Anirudh Ramanathan created SPARK-23200: -- Summary: Reset configuration when restarting from checkpoints Key: SPARK-23200 URL: https://issues.apache.org/jira/browse/SPARK-23200 Project: Spark

[jira] [Commented] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331153#comment-16331153 ] Anirudh Ramanathan commented on SPARK-22962: I think this isn't super critical for this

[jira] [Updated] (SPARK-23146) Support client mode for Kubernetes cluster backend

[ https://issues.apache.org/jira/browse/SPARK-23146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23146: --- Target Version/s: 2.4.0 > Support client mode for Kubernetes cluster backend >

[jira] [Updated] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22962: --- Affects Version/s: (was: 2.4.0) 2.3.0 > Kubernetes app

[jira] [Updated] (SPARK-23146) Support client mode for Kubernetes cluster backend

[ https://issues.apache.org/jira/browse/SPARK-23146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-23146: --- Affects Version/s: (was: 2.4.0) 2.3.0 > Support client

[jira] [Updated] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan updated SPARK-22962: --- Affects Version/s: (was: 2.3.0) 2.4.0 > Kubernetes app

[jira] [Commented] (SPARK-22962) Kubernetes app fails if local files are used

[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331085#comment-16331085 ] Anirudh Ramanathan commented on SPARK-22962: This is the resource staging server use-case. 

[jira] [Resolved] (SPARK-23016) Spark UI access and documentation

[ https://issues.apache.org/jira/browse/SPARK-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anirudh Ramanathan resolved SPARK-23016. Resolution: Fixed This is resolved and we've verified it for 2.3.0. > Spark UI

[jira] [Comment Edited] (SPARK-23082) Allow separate node selectors for driver and executors in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331081#comment-16331081 ] Anirudh Ramanathan edited comment on SPARK-23082 at 1/18/18 7:52 PM: -

[jira] [Commented] (SPARK-23082) Allow separate node selectors for driver and executors in Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331081#comment-16331081 ] Anirudh Ramanathan commented on SPARK-23082: This is an interesting feature request. We had

[jira] [Commented] (SPARK-23133) Spark options are not passed to the Executor in Docker context

[ https://issues.apache.org/jira/browse/SPARK-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331074#comment-16331074 ] Anirudh Ramanathan commented on SPARK-23133: Thanks for submitting the PR fixing this. >

[jira] [Commented] (SPARK-23016) Spark UI access and documentation

[ https://issues.apache.org/jira/browse/SPARK-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330389#comment-16330389 ] Anirudh Ramanathan commented on SPARK-23016: Good point - yeah, we don't recommend the API

  1   2   >