[jira] [Created] (SPARK-27373) Design: Kubernetes support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27373: - Summary: Design: Kubernetes support for GPU-aware scheduling Key: SPARK-27373 URL: https://issues.apache.org/jira/browse/SPARK-27373 Project: Spark Issue

[jira] [Updated] (SPARK-27362) Kubernetes support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27362: -- Description: Design and implement k8s support for GPU-aware scheduling. > Kubernetes support

[jira] [Updated] (SPARK-27360) Standalone cluster mode support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27360: -- Description: Design and implement standalone manager support for GPU-aware scheduling: 1.

[jira] [Created] (SPARK-27372) Standalone executor process-level isolation to support GPU scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27372: - Summary: Standalone executor process-level isolation to support GPU scheduling Key: SPARK-27372 URL: https://issues.apache.org/jira/browse/SPARK-27372 Project:

[jira] [Created] (SPARK-27371) Standalone worker can auto discover GPUs

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27371: - Summary: Standalone worker can auto discover GPUs Key: SPARK-27371 URL: https://issues.apache.org/jira/browse/SPARK-27371 Project: Spark Issue Type:

[jira] [Created] (SPARK-27370) spark-submit requests GPUs in standalone mode

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27370: - Summary: spark-submit requests GPUs in standalone mode Key: SPARK-27370 URL: https://issues.apache.org/jira/browse/SPARK-27370 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27360) Standalone cluster mode support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27360: -- Description: Design work for the standalone manager to support GPU-aware scheduling: 1.

[jira] [Created] (SPARK-27369) Standalone support static conf to describe GPU resources

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27369: - Summary: Standalone support static conf to describe GPU resources Key: SPARK-27369 URL: https://issues.apache.org/jira/browse/SPARK-27369 Project: Spark

[jira] [Created] (SPARK-27368) Design: Standalone supports GPU scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27368: - Summary: Design: Standalone supports GPU scheduling Key: SPARK-27368 URL: https://issues.apache.org/jira/browse/SPARK-27368 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-27360) Standalone cluster mode support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27360: - Assignee: Xiangrui Meng > Standalone cluster mode support for GPU-aware scheduling >

[jira] [Resolved] (SPARK-27005) Design sketch for SPIP discussion: Accelerator-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-27005. --- Resolution: Done Fix Version/s: 3.0.0 I'm closing the ticket since SPIP passed vote.

[jira] [Updated] (SPARK-27005) Design sketch for SPIP discussion: Accelerator-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27005: -- Summary: Design sketch for SPIP discussion: Accelerator-aware scheduling (was: Design

[jira] [Created] (SPARK-27366) Spark scheduler internal changes to support GPU scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27366: - Summary: Spark scheduler internal changes to support GPU scheduling Key: SPARK-27366 URL: https://issues.apache.org/jira/browse/SPARK-27366 Project: Spark

[jira] [Assigned] (SPARK-27005) Design sketch: Accelerator-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27005: - Assignee: Xingbo Jiang > Design sketch: Accelerator-aware scheduling >

[jira] [Updated] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27364: -- Description: Design and implement: * General guidelines for cluster managers to understand

[jira] [Updated] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27364: -- Summary: User-facing APIs for GPU-aware scheduling (was: Design: User-facing APIs for

[jira] [Updated] (SPARK-27361) YARN support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27361: -- Summary: YARN support for GPU-aware scheduling (was: Design: YARN support for GPU-aware

[jira] [Updated] (SPARK-27363) Mesos support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27363: -- Summary: Mesos support for GPU-aware scheduling (was: Design: Mesos support for GPU-aware

[jira] [Updated] (SPARK-27362) Kubernetes support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27362: -- Summary: Kubernetes support for GPU-aware scheduling (was: Design: Kubernetes support for

[jira] [Updated] (SPARK-27024) Executor interface for cluster managers to support GPU resources

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27024: -- Summary: Executor interface for cluster managers to support GPU resources (was: Design:

[jira] [Updated] (SPARK-27360) Standalone cluster mode support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27360: -- Summary: Standalone cluster mode support for GPU-aware scheduling (was: Design: Standalone

[jira] [Created] (SPARK-27365) Spark Jenkins setup to support GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27365: - Summary: Spark Jenkins setup to support GPU-aware scheduling Key: SPARK-27365 URL: https://issues.apache.org/jira/browse/SPARK-27365 Project: Spark Issue

[jira] [Updated] (SPARK-27024) Design: Executor interface for cluster managers to support GPU resources

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27024: -- Summary: Design: Executor interface for cluster managers to support GPU resources (was:

[jira] [Created] (SPARK-27364) Design: User-facing APIs for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27364: - Summary: Design: User-facing APIs for GPU-aware scheduling Key: SPARK-27364 URL: https://issues.apache.org/jira/browse/SPARK-27364 Project: Spark Issue

[jira] [Created] (SPARK-27363) Design: Mesos support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27363: - Summary: Design: Mesos support for GPU-aware scheduling Key: SPARK-27363 URL: https://issues.apache.org/jira/browse/SPARK-27363 Project: Spark Issue Type:

[jira] [Created] (SPARK-27362) Design: Kubernetes support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27362: - Summary: Design: Kubernetes support for GPU-aware scheduling Key: SPARK-27362 URL: https://issues.apache.org/jira/browse/SPARK-27362 Project: Spark Issue

[jira] [Created] (SPARK-27361) Design: YARN support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27361: - Summary: Design: YARN support for GPU-aware scheduling Key: SPARK-27361 URL: https://issues.apache.org/jira/browse/SPARK-27361 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-27024) Design executor interface to support GPU resources

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-27024: - Assignee: Thomas Graves > Design executor interface to support GPU resources >

[jira] [Created] (SPARK-27360) Design: Standalone cluster mode support for GPU-aware scheduling

2019-04-03 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27360: - Summary: Design: Standalone cluster mode support for GPU-aware scheduling Key: SPARK-27360 URL: https://issues.apache.org/jira/browse/SPARK-27360 Project: Spark

[jira] [Commented] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

2019-04-03 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809291#comment-16809291 ] Xiangrui Meng commented on SPARK-27299: --- [~mju] Could you post the draft design doc to JIRA? >

[jira] [Created] (SPARK-27317) New graph component user guide (algorithms)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27317: - Summary: New graph component user guide (algorithms) Key: SPARK-27317 URL: https://issues.apache.org/jira/browse/SPARK-27317 Project: Spark Issue Type:

[jira] [Created] (SPARK-27316) Query-based PropertyGraph algorithms

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27316: - Summary: Query-based PropertyGraph algorithms Key: SPARK-27316 URL: https://issues.apache.org/jira/browse/SPARK-27316 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27315) GraphX-based PropertyGraph algorithms

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27315: - Summary: GraphX-based PropertyGraph algorithms Key: SPARK-27315 URL: https://issues.apache.org/jira/browse/SPARK-27315 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27313) DataFrame-based PropertyGraph algorithms

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27313: - Summary: DataFrame-based PropertyGraph algorithms Key: SPARK-27313 URL: https://issues.apache.org/jira/browse/SPARK-27313 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27312) PropertyGraph <-> GraphX conversions

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27312: - Summary: PropertyGraph <-> GraphX conversions Key: SPARK-27312 URL: https://issues.apache.org/jira/browse/SPARK-27312 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27309) PropertyGraph SparkSession adapter (Scala/Java)

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27309: -- Description: Create SparkSession adapter to simply ProperGraph creation (Scala implicits?

[jira] [Updated] (SPARK-27309) PropertyGraph SparkSession adapter (Scala/Java)

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27309: -- Summary: PropertyGraph SparkSession adapter (Scala/Java) (was: ProperGraph SparkSession

[jira] [Updated] (SPARK-27300) Create the new graph projects in Spark and set up build/test

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27300: -- Description: * Create graph projects in Spark repo that works with both maven and sbt. * Add

[jira] [Created] (SPARK-27311) New graph component user guide (PropertyGraph / Query)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27311: - Summary: New graph component user guide (PropertyGraph / Query) Key: SPARK-27311 URL: https://issues.apache.org/jira/browse/SPARK-27311 Project: Spark

[jira] [Created] (SPARK-27310) PropertyGraph SparkSession adapter (Python)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27310: - Summary: PropertyGraph SparkSession adapter (Python) Key: SPARK-27310 URL: https://issues.apache.org/jira/browse/SPARK-27310 Project: Spark Issue Type:

[jira] [Created] (SPARK-27309) ProperGraph SparkSession adapter (Scala/Java)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27309: - Summary: ProperGraph SparkSession adapter (Scala/Java) Key: SPARK-27309 URL: https://issues.apache.org/jira/browse/SPARK-27309 Project: Spark Issue Type:

[jira] [Created] (SPARK-27307) PropertyGraph save/load (Python)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27307: - Summary: PropertyGraph save/load (Python) Key: SPARK-27307 URL: https://issues.apache.org/jira/browse/SPARK-27307 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27308) PropertyGraph Cypher query (Python)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27308: - Summary: PropertyGraph Cypher query (Python) Key: SPARK-27308 URL: https://issues.apache.org/jira/browse/SPARK-27308 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27306) PropertyGraph construction (Python)

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27306: -- Description: Python API and test for SPARK-27303. (was: Python API for SPARK-27303.) >

[jira] [Created] (SPARK-27306) PropertyGraph construction (Python)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27306: - Summary: PropertyGraph construction (Python) Key: SPARK-27306 URL: https://issues.apache.org/jira/browse/SPARK-27306 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-25994) SPIP: Property Graphs, Cypher Queries, and Algorithms

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25994: -- Epic Name: Property Graph / Cypher / Algorithms > SPIP: Property Graphs, Cypher Queries, and

[jira] [Created] (SPARK-27305) PropertyGraph Cypher query (Java/Scala)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27305: - Summary: PropertyGraph Cypher query (Java/Scala) Key: SPARK-27305 URL: https://issues.apache.org/jira/browse/SPARK-27305 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27300) Create the new graph projects in Spark and set up build/test

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27300: -- Summary: Create the new graph projects in Spark and set up build/test (was: Create the new

[jira] [Created] (SPARK-27304) PropertyGraph save/load (Scala/Java)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27304: - Summary: PropertyGraph save/load (Scala/Java) Key: SPARK-27304 URL: https://issues.apache.org/jira/browse/SPARK-27304 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27303) PropertyGraph construction (Scala/Java)

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27303: - Summary: PropertyGraph construction (Scala/Java) Key: SPARK-27303 URL: https://issues.apache.org/jira/browse/SPARK-27303 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27302) Design: graph algorithms based on property graphs

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27302: - Summary: Design: graph algorithms based on property graphs Key: SPARK-27302 URL: https://issues.apache.org/jira/browse/SPARK-27302 Project: Spark Issue

[jira] [Created] (SPARK-27300) Create the new graph projects in Spark

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27300: - Summary: Create the new graph projects in Spark Key: SPARK-27300 URL: https://issues.apache.org/jira/browse/SPARK-27300 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27299: -- Description: Design doc for property graph and Cypher queries. * Construct a property graph.

[jira] [Updated] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27299: -- Description: Design doc for property graph and Cypher queries. * Construct a property graph.

[jira] [Created] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

2019-03-28 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-27299: - Summary: Design: Property graph construction, save/load, and query APIs Key: SPARK-27299 URL: https://issues.apache.org/jira/browse/SPARK-27299 Project: Spark

[jira] [Updated] (SPARK-25994) SPIP: Property Graphs, Cypher Queries, and Algorithms

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25994: -- Issue Type: Epic (was: New Feature) > SPIP: Property Graphs, Cypher Queries, and Algorithms

[jira] [Updated] (SPARK-25994) SPIP: Property Graphs, Cypher Queries, and Algorithms

2019-03-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25994: -- Component/s: (was: GraphX) Graph > SPIP: Property Graphs, Cypher

[jira] [Updated] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2019-02-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Description: (The JIRA received a major update on 2019/02/28. Some comments were based on an

[jira] [Updated] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2019-02-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Description: (The JIRA received a major update on 2019/02/28. Some comments were based on an

[jira] [Updated] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2019-02-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Description: (The JIRA received a major update on 2019/02/28.) h2. Background and Motivation

[jira] [Updated] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2019-02-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Description: (The JIRA received a major update on 2019/02/28. h2. Background and Motivation

[jira] [Updated] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2019-02-27 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Summary: SPIP: Accelerator-aware task scheduling for Spark (was: Accelerator-aware task

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2019-02-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778660#comment-16778660 ] Xiangrui Meng commented on SPARK-24615: --- Attached the PDF files. > Accelerator-aware task

[jira] [Updated] (SPARK-24615) Accelerator-aware task scheduling for Spark

2019-02-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Attachment: SPIP_ Accelerator-aware scheduling.pdf > Accelerator-aware task scheduling for

[jira] [Updated] (SPARK-24615) Accelerator-aware task scheduling for Spark

2019-02-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Attachment: Accelerator-aware scheduling in Apache Spark 3.0.pdf > Accelerator-aware task

[jira] [Assigned] (SPARK-24615) Accelerator-aware task scheduling for Spark

2019-02-22 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24615: - Assignee: Xingbo Jiang > Accelerator-aware task scheduling for Spark >

[jira] [Comment Edited] (SPARK-24374) SPIP: Support Barrier Execution Mode in Apache Spark

2019-02-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766550#comment-16766550 ] Xiangrui Meng edited comment on SPARK-24374 at 2/12/19 10:54 PM: -

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Execution Mode in Apache Spark

2019-02-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766550#comment-16766550 ] Xiangrui Meng commented on SPARK-24374: --- [~luzengxiang] When you create external processes, did

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2019-01-25 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752462#comment-16752462 ] Xiangrui Meng commented on SPARK-24579: --- [~bryanc] If we limit the scope to model inference w/

[jira] [Commented] (SPARK-26410) Support per Pandas UDF configuration

2019-01-25 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752467#comment-16752467 ] Xiangrui Meng commented on SPARK-26410: --- There are several possible solutions to this. SPARK-23258

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Execution Mode in Apache Spark

2019-01-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739565#comment-16739565 ] Xiangrui Meng commented on SPARK-24374: --- [~luzengxiang] Did you use Python API or Scala API for

[jira] [Resolved] (SPARK-25970) Add Instrumentation to PrefixSpan

2018-12-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25970. --- Resolution: Done > Add Instrumentation to PrefixSpan > - >

[jira] [Assigned] (SPARK-25970) Add Instrumentation to PrefixSpan

2018-12-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25970: - Assignee: zhengruifeng > Add Instrumentation to PrefixSpan >

[jira] [Commented] (SPARK-26410) Support per Pandas UDF configuration

2018-12-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726155#comment-16726155 ] Xiangrui Meng commented on SPARK-26410: --- On the same cluster, there could be different workloads.

[jira] [Created] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames for the entire partition

2018-12-19 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-26412: - Summary: Allow Pandas UDF to take an iterator of pd.DataFrames for the entire partition Key: SPARK-26412 URL: https://issues.apache.org/jira/browse/SPARK-26412

[jira] [Updated] (SPARK-26410) Support per Pandas UDF configuration

2018-12-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-26410: -- Description: We use a "maxRecordsPerBatch" conf to control the batch sizes. However, the

[jira] [Created] (SPARK-26410) Support per Pandas UDF configuration

2018-12-19 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-26410: - Summary: Support per Pandas UDF configuration Key: SPARK-26410 URL: https://issues.apache.org/jira/browse/SPARK-26410 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2018-11-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699339#comment-16699339 ] Xiangrui Meng commented on SPARK-26175: --- This affects Hydrogen because the external training

[jira] [Updated] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2018-11-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-26175: -- Labels: Hydrogen (was: ) > PySpark cannot terminate worker process if user program reads

[jira] [Updated] (SPARK-25994) SPIP: DataFrame-based graph queries and algorithms

2018-11-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25994: -- Description: Copied from the SPIP doc: {quote} GraphX was one of the foundational pillars of

[jira] [Commented] (SPARK-25994) SPIP: DataFrame-based graph queries and algorithms

2018-11-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684524#comment-16684524 ] Xiangrui Meng commented on SPARK-25994: --- [~mju] Could you link the SPIP doc here and link the

[jira] [Created] (SPARK-26028) Design sketch for SPIP: DataFrame-based graph queries and algorithms

2018-11-12 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-26028: - Summary: Design sketch for SPIP: DataFrame-based graph queries and algorithms Key: SPARK-26028 URL: https://issues.apache.org/jira/browse/SPARK-26028 Project:

[jira] [Assigned] (SPARK-25994) SPIP: DataFrame-based graph queries and algorithms

2018-11-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25994: - Assignee: Martin Junghanns > SPIP: DataFrame-based graph queries and algorithms >

[jira] [Assigned] (SPARK-25994) SPIP: DataFrame-based graph queries and algorithms

2018-11-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-25994: - Assignee: (was: Xiangrui Meng) > SPIP: DataFrame-based graph queries and

[jira] [Created] (SPARK-25994) SPIP: DataFrame-based graph queries and algorithms

2018-11-09 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-25994: - Summary: SPIP: DataFrame-based graph queries and algorithms Key: SPARK-25994 URL: https://issues.apache.org/jira/browse/SPARK-25994 Project: Spark Issue

[jira] [Commented] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672350#comment-16672350 ] Xiangrui Meng commented on SPARK-25921: --- Tested offline. This was caused by PySpark worker reuse.

[jira] [Updated] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25921: -- Priority: Critical (was: Major) > Python worker reuse causes Barrier tasks to run without

[jira] [Updated] (SPARK-25921) Python worker reuse causes Barrier tasks to run without BarrierTaskContext

2018-11-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25921: -- Target Version/s: 2.4.1 > Python worker reuse causes Barrier tasks to run without

[jira] [Updated] (SPARK-25793) Loading model bug in BisectingKMeans

2018-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25793: -- Target Version/s: 2.4.1, 2.5.0 > Loading model bug in BisectingKMeans >

[jira] [Resolved] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

2018-10-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-14681. --- Resolution: Won't Do Fix Version/s: (was: 2.4.0) Marked the ticket as "won't do"

[jira] [Commented] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

2018-10-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636138#comment-16636138 ] Xiangrui Meng commented on SPARK-14681: --- The change were reverted in both branch-2.4 and master to

[jira] [Reopened] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

2018-10-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reopened SPARK-14681: --- > Provide label/impurity stats for spark.ml decision tree nodes >

[jira] [Resolved] (SPARK-25322) ML, Graph 2.4 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25322. --- Resolution: Done > ML, Graph 2.4 QA: API: Experimental, DeveloperApi, final, sealed audit >

[jira] [Resolved] (SPARK-25319) Spark MLlib, GraphX 2.4 QA umbrella

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25319. --- Resolution: Done Assignee: Weichen Xu (was: Joseph K. Bradley) > Spark MLlib, GraphX

[jira] [Resolved] (SPARK-25325) ML, Graph 2.4 QA: Update user guide for new features & APIs

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25325. --- Resolution: Won't Do > ML, Graph 2.4 QA: Update user guide for new features & APIs >

[jira] [Updated] (SPARK-25323) ML 2.4 QA: API: Python API coverage

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25323: -- Priority: Major (was: Critical) > ML 2.4 QA: API: Python API coverage >

[jira] [Resolved] (SPARK-25323) ML 2.4 QA: API: Python API coverage

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25323. --- Resolution: Won't Do > ML 2.4 QA: API: Python API coverage >

[jira] [Updated] (SPARK-25325) ML, Graph 2.4 QA: Update user guide for new features & APIs

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25325: -- Priority: Major (was: Critical) > ML, Graph 2.4 QA: Update user guide for new features &

[jira] [Resolved] (SPARK-25326) ML, Graph 2.4 QA: Programming guide update and migration guide

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-25326. --- Resolution: Won't Do > ML, Graph 2.4 QA: Programming guide update and migration guide >

[jira] [Updated] (SPARK-25326) ML, Graph 2.4 QA: Programming guide update and migration guide

2018-10-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-25326: -- Priority: Major (was: Critical) > ML, Graph 2.4 QA: Programming guide update and migration

<    1   2   3   4   5   6   7   8   9   10   >