[jira] [Commented] (SPARK-24725) Discuss necessary info and access in barrier mode + Mesos

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530627#comment-16530627 ] Xiangrui Meng commented on SPARK-24725: --- [~dbtsai] Apple Siri is a major user of Spark + Mesos. Do

[jira] [Updated] (SPARK-24726) Discuss necessary info and access in barrier mode + Standalone

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24726: -- Description: In barrier mode, to run hybrid distributed DL training jobs, we need to provide

[jira] [Created] (SPARK-24726) Discuss necessary info and access in barrier mode + Standalone

2018-07-02 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24726: - Summary: Discuss necessary info and access in barrier mode + Standalone Key: SPARK-24726 URL: https://issues.apache.org/jira/browse/SPARK-24726 Project: Spark

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Shepherd: Xiangrui Meng > Accelerator aware task scheduling for Spark >

[jira] [Created] (SPARK-24725) Discuss necessary info and access in barrier mode + Mesos

2018-07-02 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24725: - Summary: Discuss necessary info and access in barrier mode + Mesos Key: SPARK-24725 URL: https://issues.apache.org/jira/browse/SPARK-24725 Project: Spark

[jira] [Updated] (SPARK-24725) Discuss necessary info and access in barrier mode + Mesos

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24725: -- Description: In barrier mode, to run hybrid distributed DL training jobs, we need to provide

[jira] [Created] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

2018-07-02 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24724: - Summary: Discuss necessary info and access in barrier mode + Kubernetes Key: SPARK-24724 URL: https://issues.apache.org/jira/browse/SPARK-24724 Project: Spark

[jira] [Updated] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24724: -- Description: In barrier mode, to run hybrid distributed DL training jobs, we need to provide

[jira] [Created] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

2018-07-02 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24723: - Summary: Discuss necessary info and access in barrier mode + YARN Key: SPARK-24723 URL: https://issues.apache.org/jira/browse/SPARK-24723 Project: Spark

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530563#comment-16530563 ] Xiangrui Meng commented on SPARK-24579: --- Addressed the inline comments in the doc. I'm going to

[jira] [Updated] (SPARK-24719) ClusteringEvaluator supports integer type labels

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24719: -- Description: ClusterEvaluator should support integer labels because we output integer labels

[jira] [Updated] (SPARK-24719) ClusteringEvaluator supports integer type labels

2018-07-02 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24719: -- Description: ClusterEvaluator should support integer labels because we output integer labels

[jira] [Created] (SPARK-24719) ClusteringEvaluator supports integer type labels

2018-07-02 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24719: - Summary: ClusteringEvaluator supports integer type labels Key: SPARK-24719 URL: https://issues.apache.org/jira/browse/SPARK-24719 Project: Spark Issue

[jira] [Assigned] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-28 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24530: - Assignee: Hyukjin Kwon > Sphinx doesn't render autodoc_docstring_signature correctly

[jira] [Updated] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

2018-06-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Summary: Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and

[jira] [Updated] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (using Python 2?)

2018-06-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Summary: Sphinx doesn't render autodoc_docstring_signature correctly (using Python 2?) (was:

[jira] [Commented] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524510#comment-16524510 ] Xiangrui Meng commented on SPARK-24530: --- Confirmed that macOS, python 3, and Sphinx v1.6.6 can

[jira] [Commented] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-25 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522661#comment-16522661 ] Xiangrui Meng commented on SPARK-24530: --- [~dongjoon] [~hyukjin.kwon] Could you report your system,

[jira] [Updated] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-21 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Labels: Hydrogen SPIP (was: SPIP) > Accelerator aware task scheduling for Spark >

[jira] [Updated] (SPARK-24609) PySpark doc doesn't explain RandomForestClassifier.featureSubsetStrategy well

2018-06-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24609: -- Description: In Scala doc

[jira] [Updated] (SPARK-24609) PySpark/SparkR doc doesn't explain RandomForestClassifier.featureSubsetStrategy well

2018-06-20 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24609: -- Summary: PySpark/SparkR doc doesn't explain RandomForestClassifier.featureSubsetStrategy well

[jira] [Created] (SPARK-24609) PySpark doc doesn't explain RandomForestClassifier.featureSubsetStrategy well

2018-06-20 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24609: - Summary: PySpark doc doesn't explain RandomForestClassifier.featureSubsetStrategy well Key: SPARK-24609 URL: https://issues.apache.org/jira/browse/SPARK-24609

[jira] [Resolved] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2018-06-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-12436. --- Resolution: Resolved Resolved by SPARK-23772 (an alternative solution to this JIRA) > If

[jira] [Resolved] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24375. --- Resolution: Done > Design sketch: support barrier scheduling in Apache Spark >

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515913#comment-16515913 ] Xiangrui Meng commented on SPARK-24375: --- I'm closing this Jira in favor of formal design

[jira] [Assigned] (SPARK-24580) List scenarios to be handled by barrier execution mode properly

2018-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24580: - Assignee: Jiang Xingbo (was: Xiangrui Meng) > List scenarios to be handled by barrier

[jira] [Updated] (SPARK-24580) List scenarios to be handled by barrier execution mode properly

2018-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24580: -- Description: List scenarios to be handled by barrier execution mode to help the design. We

[jira] [Created] (SPARK-24582) Design: Barrier execution mode

2018-06-18 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24582: - Summary: Design: Barrier execution mode Key: SPARK-24582 URL: https://issues.apache.org/jira/browse/SPARK-24582 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-24581) Design: BarrierTaskContext.barrier()

2018-06-18 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24581: - Summary: Design: BarrierTaskContext.barrier() Key: SPARK-24581 URL: https://issues.apache.org/jira/browse/SPARK-24581 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-24580) List scenarios to be handled by barrier execution mode properly

2018-06-18 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24580: - Summary: List scenarios to be handled by barrier execution mode properly Key: SPARK-24580 URL: https://issues.apache.org/jira/browse/SPARK-24580 Project: Spark

[jira] [Updated] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24579: -- Labels: Hydrogen (was: ) > SPIP: Standardize Optimized Data Exchange between Spark and DL/AI

[jira] [Updated] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24579: -- Description: (see attached SPIP pdf for more details) At the crossroads of big data and AI,

[jira] [Updated] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-06-18 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24579: -- Attachment: [SPARK-24579] SPIP_ Standardize Optimized Data Exchange between Apache Spark and

[jira] [Created] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-06-18 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24579: - Summary: SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks Key: SPARK-24579 URL: https://issues.apache.org/jira/browse/SPARK-24579

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Description: I generated python docs from master locally using `make html`. However, the

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Priority: Blocker (was: Major) > pyspark.ml doesn't generate class docs correctly >

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Attachment: Screen Shot 2018-06-12 at 8.23.18 AM.png > pyspark.ml doesn't generate class docs

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Attachment: Screen Shot 2018-06-12 at 8.23.29 AM.png > pyspark.ml doesn't generate class docs

[jira] [Created] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24530: - Summary: pyspark.ml doesn't generate class docs correctly Key: SPARK-24530 URL: https://issues.apache.org/jira/browse/SPARK-24530 Project: Spark Issue

[jira] [Resolved] (SPARK-15064) Locale support in StopWordsRemover

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-15064. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21501

[jira] [Resolved] (SPARK-19826) spark.ml Python API for PIC

2018-06-11 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-19826. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21513

[jira] [Updated] (SPARK-22666) Spark datasource for image format

2018-06-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-22666: -- Target Version/s: 2.4.0 > Spark datasource for image format >

[jira] [Commented] (SPARK-22666) Spark datasource for image format

2018-06-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506471#comment-16506471 ] Xiangrui Meng commented on SPARK-22666: --- [~mhamilton] Do you want to take this task? > Spark

[jira] [Updated] (SPARK-22666) Spark datasource for image format

2018-06-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-22666: -- Shepherd: Xiangrui Meng > Spark datasource for image format >

[jira] [Resolved] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24454. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21483

[jira] [Assigned] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24454: - Assignee: Hyukjin Kwon > ml.image doesn't have __all__ explicitly defined >

[jira] [Assigned] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24477: - Assignee: Hyukjin Kwon > Import submodules under pyspark.ml by default >

[jira] [Resolved] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24477. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21483

[jira] [Issue Comment Deleted] (SPARK-15064) Locale support in StopWordsRemover

2018-06-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-15064: -- Comment: was deleted (was: Yuhao will be OOF from May 29th to June 6th (annual leave and

[jira] [Updated] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24477: -- Description: Right now, we do not import submodules under pyspark.ml by default. So users

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Issue Type: Improvement (was: Bug) > ml.image doesn't have __all__ explicitly defined >

[jira] [Commented] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503700#comment-16503700 ] Xiangrui Meng commented on SPARK-24454: --- Updated this JIRA and created SPARK-24477. > ml.image

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Priority: Minor (was: Major) > ml.image doesn't have __all__ explicitly defined >

[jira] [Created] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24477: - Summary: Import submodules under pyspark.ml by default Key: SPARK-24477 URL: https://issues.apache.org/jira/browse/SPARK-24477 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Description: ml/image.py doesn't have __all__ explicitly defined. It will import all global

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Summary: ml.image doesn't have __all__ explicitly defined (was: ml.image doesn't have

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-06-05 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Labels: Hydrogen SPIP (was: SPIP) > SPIP: Support Barrier Scheduling in Apache Spark >

[jira] [Updated] (SPARK-19826) spark.ml Python API for PIC

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-19826: -- Shepherd: Weichen Xu > spark.ml Python API for PIC > --- > >

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Epic Name: Support Barrier Execution Mode > SPIP: Support Barrier Scheduling in Apache Spark

[jira] [Assigned] (SPARK-19826) spark.ml Python API for PIC

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-19826: - Assignee: Huaxin Gao > spark.ml Python API for PIC > --- > >

[jira] [Resolved] (SPARK-15784) Add Power Iteration Clustering to spark.ml

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-15784. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21493

[jira] [Resolved] (SPARK-24300) generateLDAData in ml.cluster.LDASuite didn't set seed correctly

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24300. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21492

[jira] [Created] (SPARK-24464) Unit tests for MLlib's Instrumentation

2018-06-04 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24464: - Summary: Unit tests for MLlib's Instrumentation Key: SPARK-24464 URL: https://issues.apache.org/jira/browse/SPARK-24464 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-24290) Instrumentation Improvement: add logNamedValue taking Array types

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24290. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21347

[jira] [Assigned] (SPARK-24290) Instrumentation Improvement: add logNamedValue taking Array types

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24290: - Assignee: Lu Wang > Instrumentation Improvement: add logNamedValue taking Array types

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500900#comment-16500900 ] Xiangrui Meng commented on SPARK-24374: --- [~leftnoteasy] Thanks for your input! {quote} This JIRA

[jira] [Commented] (SPARK-15784) Add Power Iteration Clustering to spark.ml

2018-06-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500809#comment-16500809 ] Xiangrui Meng commented on SPARK-15784: --- Discussed with [~WeichenXu123] offline. I think we should

[jira] [Updated] (SPARK-24454) ml.image doesn't have default imports

2018-06-01 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Description: {code:java} from pyspark import ml ml.image{code} The ml.image line will fail

[jira] [Created] (SPARK-24454) ml.image doesn't have default imports

2018-06-01 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24454: - Summary: ml.image doesn't have default imports Key: SPARK-24454 URL: https://issues.apache.org/jira/browse/SPARK-24454 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-24146) spark.ml parity for sequential pattern mining - PrefixSpan: Python API

2018-05-31 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24146. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21265

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-24 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489333#comment-16489333 ] Xiangrui Meng commented on SPARK-24374: --- [~galv] Thanks for your feedback! * SPARK-20327 allows

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488432#comment-16488432 ] Xiangrui Meng commented on SPARK-24375: --- [~jiangxb1987] Could you summarize the design sketch based

[jira] [Created] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24375: - Summary: Design sketch: support barrier scheduling in Apache Spark Key: SPARK-24375 URL: https://issues.apache.org/jira/browse/SPARK-24375 Project: Spark

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Shepherd: Reynold Xin > SPIP: Support Barrier Scheduling in Apache Spark >

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Issue Type: Epic (was: Story) > SPIP: Support Barrier Scheduling in Apache Spark >

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Description: (See details in the linked/attached SPIP doc.) {quote} The proposal here is to

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Attachment: SPIP_ Support Barrier Scheduling in Apache Spark.pdf > SPIP: Support Barrier

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Description: (See details in the linked/attached SPIP doc.) The proposal here is to add a new

[jira] [Updated] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24374: -- Labels: SPIP (was: ) > SPIP: Support Barrier Scheduling in Apache Spark >

[jira] [Created] (SPARK-24374) SPIP: Support Barrier Scheduling in Apache Spark

2018-05-23 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24374: - Summary: SPIP: Support Barrier Scheduling in Apache Spark Key: SPARK-24374 URL: https://issues.apache.org/jira/browse/SPARK-24374 Project: Spark Issue

[jira] [Resolved] (SPARK-22884) ML test for StructuredStreaming: spark.ml.clustering

2018-05-17 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-22884. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21358

[jira] [Resolved] (SPARK-24115) improve instrumentation for spark.ml.tuning

2018-05-17 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24115. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21340

[jira] [Assigned] (SPARK-24115) improve instrumentation for spark.ml.tuning

2018-05-17 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24115: - Assignee: Bago Amirbekian > improve instrumentation for spark.ml.tuning >

[jira] [Assigned] (SPARK-24300) generateLDAData in ml.cluster.LDASuite didn't set seed correctly

2018-05-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24300: - Assignee: Lu Wang > generateLDAData in ml.cluster.LDASuite didn't set seed correctly >

[jira] [Created] (SPARK-24300) generateLDAData in ml.cluster.LDASuite didn't set seed correctly

2018-05-16 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24300: - Summary: generateLDAData in ml.cluster.LDASuite didn't set seed correctly Key: SPARK-24300 URL: https://issues.apache.org/jira/browse/SPARK-24300 Project: Spark

[jira] [Assigned] (SPARK-24146) spark.ml parity for sequential pattern mining - PrefixSpan: Python API

2018-05-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24146: - Assignee: Weichen Xu > spark.ml parity for sequential pattern mining - PrefixSpan:

[jira] [Resolved] (SPARK-24231) Python API: Provide evaluateEachIteration method or equivalent for spark.ml GBTs

2018-05-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24231. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21335

[jira] [Assigned] (SPARK-24231) Python API: Provide evaluateEachIteration method or equivalent for spark.ml GBTs

2018-05-15 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24231: - Assignee: Lu Wang > Python API: Provide evaluateEachIteration method or equivalent for

[jira] [Assigned] (SPARK-24155) Instrumentation improvement for clustering

2018-05-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24155: - Assignee: Lu Wang > Instrumentation improvement for clustering >

[jira] [Resolved] (SPARK-24155) Instrumentation improvement for clustering

2018-05-14 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24155. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21218

[jira] [Assigned] (SPARK-24132) Instrumentation improvement for classification

2018-05-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24132: - Assignee: Lu Wang > Instrumentation improvement for classification >

[jira] [Resolved] (SPARK-24132) Instrumentation improvement for classification

2018-05-08 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-24132. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 21204

[jira] [Resolved] (SPARK-23975) Allow Clustering to take Arrays of Double as input features

2018-05-07 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-23975. --- Resolution: Fixed Fix Version/s: 2.4.0 > Allow Clustering to take Arrays of Double as

[jira] [Resolved] (SPARK-10383) Sync example code between API doc and user guide

2018-05-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10383. --- Resolution: Won't Do Target Version/s: (was: ) > Sync example code between API

[jira] [Assigned] (SPARK-18924) Improve collect/createDataFrame performance in SparkR

2018-05-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-18924: - Assignee: (was: Xiangrui Meng) > Improve collect/createDataFrame performance in

[jira] [Resolved] (SPARK-7924) Consolidate example code in MLlib

2018-05-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-7924. -- Resolution: Done > Consolidate example code in MLlib > - > >

[jira] [Resolved] (SPARK-5874) How to improve the current ML pipeline API?

2018-05-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-5874. -- Resolution: Done > How to improve the current ML pipeline API? >

[jira] [Resolved] (SPARK-12285) MLlib user guide: umbrella for missing sections

2018-05-04 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-12285. --- Resolution: Done > MLlib user guide: umbrella for missing sections >

[jira] [Assigned] (SPARK-23772) Provide an option to ignore column of all null values or empty map/array during JSON schema inference

2018-04-09 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-23772: - Assignee: Takeshi Yamamuro > Provide an option to ignore column of all null values or

[jira] [Created] (SPARK-23772) Provide an option to ignore column of all null values or empty map/array during JSON schema inference

2018-03-22 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-23772: - Summary: Provide an option to ignore column of all null values or empty map/array during JSON schema inference Key: SPARK-23772 URL:

<    1   2   3   4   5   6   7   8   9   10   >