[jira] [Commented] (SPARK-32259) tmpfs=true, not pointing to SPARK_LOCAL_DIRS in k8s

2020-07-13 Thread Rob Vesse (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156627#comment-17156627 ] Rob Vesse commented on SPARK-32259: --- bq. We use Spark launcher to do spark submit in k

[jira] [Comment Edited] (SPARK-32259) tmpfs=true, not pointing to SPARK_LOCAL_DIRS in k8s

2020-07-13 Thread Rob Vesse (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156626#comment-17156626 ] Rob Vesse edited comment on SPARK-32259 at 7/13/20, 10:32 AM:

[jira] [Commented] (SPARK-32259) tmpfs=true, not pointing to SPARK_LOCAL_DIRS in k8s

2020-07-13 Thread Rob Vesse (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156626#comment-17156626 ] Rob Vesse commented on SPARK-32259: --- [~prakki79] Ideally you'd also include the follow

[jira] [Created] (SPARK-28649) Git Ignore does not ignore python/.eggs

2019-08-07 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-28649: - Summary: Git Ignore does not ignore python/.eggs Key: SPARK-28649 URL: https://issues.apache.org/jira/browse/SPARK-28649 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2019-04-24 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825132#comment-16825132 ] Rob Vesse commented on SPARK-25262: --- [~Udbhav Agrawal] Yes I think an approach like th

[jira] [Commented] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-06 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786051#comment-16786051 ] Rob Vesse commented on SPARK-27063: --- [~skonto] Yes we have experienced the same proble

[jira] [Created] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-05 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-27063: - Summary: Spark on K8S Integration Tests timeouts are too short for some test clusters Key: SPARK-27063 URL: https://issues.apache.org/jira/browse/SPARK-27063 Project: Spark

[jira] [Commented] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements

2019-02-06 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761667#comment-16761667 ] Rob Vesse commented on SPARK-26833: --- Although not sure the latter is doable. With {{k

[jira] [Updated] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements

2019-02-06 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Vesse updated SPARK-26833: -- Description: I've seen a couple of users get bitten by this in informal discussions on GitHub and Sla

[jira] [Updated] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements

2019-02-06 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Vesse updated SPARK-26833: -- Description: I've seen a couple of users get bitten by this in informal discussions on GitHub and Sla

[jira] [Updated] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements

2019-02-06 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Vesse updated SPARK-26833: -- Description: I've seen a couple of users get bitten by this in informal discussions on GitHub and Sla

[jira] [Created] (SPARK-26833) Kubernetes RBAC documentation is unclear on exact RBAC requirements

2019-02-06 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-26833: - Summary: Kubernetes RBAC documentation is unclear on exact RBAC requirements Key: SPARK-26833 URL: https://issues.apache.org/jira/browse/SPARK-26833 Project: Spark

[jira] [Created] (SPARK-26729) Spark on Kubernetes tooling hardcodes default image names

2019-01-25 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-26729: - Summary: Spark on Kubernetes tooling hardcodes default image names Key: SPARK-26729 URL: https://issues.apache.org/jira/browse/SPARK-26729 Project: Spark Issue Typ

[jira] [Resolved] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Vesse resolved SPARK-26704. --- Resolution: Not A Problem > docker-image-tool.sh should copy custom Dockerfiles into the build conte

[jira] [Commented] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750366#comment-16750366 ] Rob Vesse commented on SPARK-26704: --- Yes sorry I'm conflating with the build context w

[jira] [Commented] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750323#comment-16750323 ] Rob Vesse commented on SPARK-26704: --- For me it's a question of build reproducibility (

[jira] [Created] (SPARK-26704) docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images

2019-01-23 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-26704: - Summary: docker-image-tool.sh should copy custom Dockerfiles into the build context for inclusion in images Key: SPARK-26704 URL: https://issues.apache.org/jira/browse/SPARK-26704

[jira] [Created] (SPARK-26687) Building Spark Images has non-intuitive behaviour with paths to custom Dockerfiles

2019-01-22 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-26687: - Summary: Building Spark Images has non-intuitive behaviour with paths to custom Dockerfiles Key: SPARK-26687 URL: https://issues.apache.org/jira/browse/SPARK-26687 Project:

[jira] [Commented] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748605#comment-16748605 ] Rob Vesse commented on SPARK-26685: --- Opened a PR to fix this > Building Spark Images

[jira] [Created] (SPARK-26685) Building Spark Images with latest Docker does not honour spark_uid build argument

2019-01-22 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-26685: - Summary: Building Spark Images with latest Docker does not honour spark_uid build argument Key: SPARK-26685 URL: https://issues.apache.org/jira/browse/SPARK-26685 Project:

[jira] [Commented] (SPARK-26028) Design sketch for SPIP: Property Graphs, Cypher Queries, and Algorithms

2019-01-18 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746134#comment-16746134 ] Rob Vesse commented on SPARK-26028: --- One initial comment - why include the query engin

[jira] [Created] (SPARK-26015) Include a USER directive in project provided Spark Dockerfiles

2018-11-12 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-26015: - Summary: Include a USER directive in project provided Spark Dockerfiles Key: SPARK-26015 URL: https://issues.apache.org/jira/browse/SPARK-26015 Project: Spark Iss

[jira] [Resolved] (SPARK-24833) Allow specifying Kubernetes host name aliases in the pod specs

2018-10-31 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Vesse resolved SPARK-24833. --- Resolution: Won't Fix As discussed in PR the pod template feature provides the ability to do this w

[jira] [Created] (SPARK-25887) Allow specifying Kubernetes context to use

2018-10-30 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-25887: - Summary: Allow specifying Kubernetes context to use Key: SPARK-25887 URL: https://issues.apache.org/jira/browse/SPARK-25887 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25809) Support additional K8S cluster types for integration tests

2018-10-30 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669048#comment-16669048 ] Rob Vesse commented on SPARK-25809: --- Fairly close to this being ready to merge > Supp

[jira] [Created] (SPARK-25809) Support additional K8S cluster types for integration tests

2018-10-23 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-25809: - Summary: Support additional K8S cluster types for integration tests Key: SPARK-25809 URL: https://issues.apache.org/jira/browse/SPARK-25809 Project: Spark Issue Ty

[jira] [Created] (SPARK-25745) docker-image-tool.sh ignores errors from Docker

2018-10-16 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-25745: - Summary: docker-image-tool.sh ignores errors from Docker Key: SPARK-25745 URL: https://issues.apache.org/jira/browse/SPARK-25745 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-23153) Support application dependencies in submission client's local file system

2018-10-03 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Vesse updated SPARK-23153: -- Description: Currently local dependencies are not supported with Spark on K8S i.e. if the user has cod

[jira] [Commented] (SPARK-23153) Support application dependencies in submission client's local file system

2018-10-02 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635534#comment-16635534 ] Rob Vesse commented on SPARK-23153: --- [~cloud_fan][~liyinan926][~mcheah][~eje] Has ther

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-09-19 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620809#comment-16620809 ] Rob Vesse commented on SPARK-24434: --- Started a mailing list thread re: the limitations

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-09-07 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606797#comment-16606797 ] Rob Vesse commented on SPARK-25262: --- [~mcheah] I would like to keep this open so we ca

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-31 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598402#comment-16598402 ] Rob Vesse commented on SPARK-24434: --- {quote} I think the miscommunication here was bec

[jira] [Commented] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-08-28 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595178#comment-16595178 ] Rob Vesse commented on SPARK-25262: --- I have changes for this almost ready and plan to

[jira] [Created] (SPARK-25262) Make Spark local dir volumes configurable with Spark on Kubernetes

2018-08-28 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-25262: - Summary: Make Spark local dir volumes configurable with Spark on Kubernetes Key: SPARK-25262 URL: https://issues.apache.org/jira/browse/SPARK-25262 Project: Spark

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-28 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594933#comment-16594933 ] Rob Vesse commented on SPARK-25024: --- [~tgraves] Attempting to answer your questions:

[jira] [Commented] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status

2018-08-24 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591386#comment-16591386 ] Rob Vesse commented on SPARK-25222: --- There is also a similar issue with task failure:

[jira] [Created] (SPARK-25222) Spark on Kubernetes Pod Watcher dumps raw container status

2018-08-24 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-25222: - Summary: Spark on Kubernetes Pod Watcher dumps raw container status Key: SPARK-25222 URL: https://issues.apache.org/jira/browse/SPARK-25222 Project: Spark Issue Ty

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-02 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567017#comment-16567017 ] Rob Vesse commented on SPARK-24434: --- [~skonto] Added a couple more comments based on s

[jira] [Created] (SPARK-24833) Allow specifying Kubernetes host name aliases in the pod specs

2018-07-17 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-24833: - Summary: Allow specifying Kubernetes host name aliases in the pod specs Key: SPARK-24833 URL: https://issues.apache.org/jira/browse/SPARK-24833 Project: Spark Iss

[jira] [Commented] (SPARK-23257) Implement Kerberos Support in Kubernetes resource manager

2018-05-31 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496284#comment-16496284 ] Rob Vesse commented on SPARK-23257: --- [~ifilonenko] Any updates on this? We're current

[jira] [Created] (SPARK-23374) Checkstyle/Scalastyle only work from top level build

2018-02-09 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-23374: - Summary: Checkstyle/Scalastyle only work from top level build Key: SPARK-23374 URL: https://issues.apache.org/jira/browse/SPARK-23374 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2017-10-20 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212462#comment-16212462 ] Rob Vesse commented on SPARK-9: --- [~yuvaldeg], thanks for providing additional clari

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2017-10-18 Thread Rob Vesse (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209396#comment-16209396 ] Rob Vesse commented on SPARK-9: --- I'd be interested to know more about what performa