[jira] [Commented] (SPARK-20607) Add new unit tests to ShuffleSuite

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997821#comment-15997821 ] Apache Spark commented on SPARK-20607: -- User 'heary-cao' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20563) going to DataFrame to RDD and back changes the schema, if the schema is not explicitly provided

2017-05-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20563. -- Resolution: Not A Problem I am resolving this per ^. Please reopen this if I misunderstood. >

[jira] [Created] (SPARK-20607) Add new unit tests to ShuffleSuite

2017-05-04 Thread caoxuewen (JIRA)
caoxuewen created SPARK-20607: - Summary: Add new unit tests to ShuffleSuite Key: SPARK-20607 URL: https://issues.apache.org/jira/browse/SPARK-20607 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-20606: --- Assignee: Yanbo Liang > ML 2.2 QA: Remove deprecated methods for ML >

[jira] [Assigned] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20606: Assignee: (was: Apache Spark) > ML 2.2 QA: Remove deprecated methods for ML >

[jira] [Assigned] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20606: Assignee: Apache Spark > ML 2.2 QA: Remove deprecated methods for ML >

[jira] [Commented] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997783#comment-15997783 ] Apache Spark commented on SPARK-20606: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20257) Fix test for directory created to work when running as R CMD check

2017-05-04 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20257. -- Resolution: Not A Problem Assignee: Felix Cheung We have disabled this test on CRAN, so

[jira] [Resolved] (SPARK-20015) Document R Structured Streaming (experimental) in R vignettes and R & SS programming guide, R example

2017-05-04 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20015. -- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 Target

[jira] [Updated] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-20606: Component/s: (was: MLlib) > ML 2.2 QA: Remove deprecated methods for ML >

[jira] [Updated] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-20606: Description: Remove ML methods we deprecated in 2.1. (was: Remove deprecated methods for ML.) >

[jira] [Updated] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-20606: Description: Remove deprecated methods for ML. > ML 2.2 QA: Remove deprecated methods for ML >

[jira] [Updated] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML

2017-05-04 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-20606: Summary: ML 2.2 QA: Remove deprecated methods for ML (was: ML 2.2 QA: Remove deprecated methods

[jira] [Assigned] (SPARK-20605) Deprecate not used AM and executor port configuration

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20605: Assignee: Apache Spark > Deprecate not used AM and executor port configuration >

[jira] [Assigned] (SPARK-20605) Deprecate not used AM and executor port configuration

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20605: Assignee: (was: Apache Spark) > Deprecate not used AM and executor port configuration

[jira] [Commented] (SPARK-20605) Deprecate not used AM and executor port configuration

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997755#comment-15997755 ] Apache Spark commented on SPARK-20605: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Created] (SPARK-20606) ML 2.2 QA: Remove deprecated methods for ML/MLlib

2017-05-04 Thread Yanbo Liang (JIRA)
Yanbo Liang created SPARK-20606: --- Summary: ML 2.2 QA: Remove deprecated methods for ML/MLlib Key: SPARK-20606 URL: https://issues.apache.org/jira/browse/SPARK-20606 Project: Spark Issue Type:

[jira] [Created] (SPARK-20605) Deprecate not used AM and executor port configuration

2017-05-04 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20605: --- Summary: Deprecate not used AM and executor port configuration Key: SPARK-20605 URL: https://issues.apache.org/jira/browse/SPARK-20605 Project: Spark Issue

[jira] [Closed] (SPARK-20574) Allow Bucketizer to handle non-Double column

2017-05-04 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang closed SPARK-20574. --- Resolution: Fixed Fix Version/s: 2.2.0 > Allow Bucketizer to handle non-Double column >

[jira] [Resolved] (SPARK-20571) Flaky SparkR StructuredStreaming tests

2017-05-04 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20571. -- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 Target

[jira] [Assigned] (SPARK-20456) Add examples for functions collection for pyspark

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20456: Assignee: (was: Apache Spark) > Add examples for functions collection for pyspark >

[jira] [Assigned] (SPARK-20456) Add examples for functions collection for pyspark

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20456: Assignee: Apache Spark > Add examples for functions collection for pyspark >

[jira] [Commented] (SPARK-20456) Add examples for functions collection for pyspark

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997720#comment-15997720 ] Apache Spark commented on SPARK-20456: -- User 'map222' has created a pull request for this issue:

[jira] [Updated] (SPARK-19878) Add hive configuration when initialize hive serde in InsertIntoHiveTable.scala

2017-05-04 Thread kavn qin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kavn qin updated SPARK-19878: - Priority: Major (was: Minor) > Add hive configuration when initialize hive serde in

[jira] [Updated] (SPARK-20456) Add examples for functions collection for pyspark

2017-05-04 Thread Michael Patterson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Patterson updated SPARK-20456: -- Description: Document sql.functions.py: 1. Add examples for the common string

[jira] [Updated] (SPARK-19690) Join a streaming DataFrame with a batch DataFrame may not work

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19690: - Target Version/s: 2.3.0 (was: 2.2.0) > Join a streaming DataFrame with a batch DataFrame may

[jira] [Comment Edited] (SPARK-20454) Improvement of ShortestPaths in Spark GraphX

2017-05-04 Thread Jennifer Le (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997549#comment-15997549 ] Jennifer Le edited comment on SPARK-20454 at 5/4/17 10:43 PM: -- Ji is right.

[jira] [Commented] (SPARK-20454) Improvement of ShortestPaths in Spark GraphX

2017-05-04 Thread Jennifer Le (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997549#comment-15997549 ] Jennifer Le commented on SPARK-20454: - Ji is right. For example, in Linkedin social network, we need

[jira] [Updated] (SPARK-20594) The staging directory should be appended with ".hive-staging" to avoid being deleted if we set hive.exec.stagingdir under the table directory without start with "."

2017-05-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-20594: Target Version/s: 2.2.0 > The staging directory should be appended with ".hive-staging" to avoid being >

[jira] [Assigned] (SPARK-20604) Allow Imputer to handle all numeric types

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20604: Assignee: (was: Apache Spark) > Allow Imputer to handle all numeric types >

[jira] [Commented] (SPARK-20604) Allow Imputer to handle all numeric types

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997533#comment-15997533 ] Apache Spark commented on SPARK-20604: -- User 'actuaryzhang' has created a pull request for this

[jira] [Assigned] (SPARK-20604) Allow Imputer to handle all numeric types

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20604: Assignee: Apache Spark > Allow Imputer to handle all numeric types >

[jira] [Updated] (SPARK-20604) Allow Imputer to handle all numeric types

2017-05-04 Thread Wayne Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wayne Zhang updated SPARK-20604: Description: Imputer currently requires input column to be Double or Float, but the logic should

[jira] [Created] (SPARK-20604) Allow Imputer to handle all numeric types

2017-05-04 Thread Wayne Zhang (JIRA)
Wayne Zhang created SPARK-20604: --- Summary: Allow Imputer to handle all numeric types Key: SPARK-20604 URL: https://issues.apache.org/jira/browse/SPARK-20604 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-20499) Spark MLlib, GraphX 2.2 QA umbrella

2017-05-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-20499: -- Description: This JIRA lists tasks for the next Spark release's QA period for MLlib

[jira] [Commented] (SPARK-20454) Improvement of ShortestPaths in Spark GraphX

2017-05-04 Thread Ji Dai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997518#comment-15997518 ] Ji Dai commented on SPARK-20454: It is a very practical issue and from real request. Would you please

[jira] [Updated] (SPARK-20454) Improvement of ShortestPaths in Spark GraphX

2017-05-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20454: -- Flags: (was: Important) Priority: Minor (was: Major) Issue Type: Improvement (was:

[jira] [Updated] (SPARK-20454) Improvement of ShortestPaths in Spark GraphX

2017-05-04 Thread Ji Dai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ji Dai updated SPARK-20454: --- Issue Type: Bug (was: Improvement) > Improvement of ShortestPaths in Spark GraphX >

[jira] [Commented] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997443#comment-15997443 ] Apache Spark commented on SPARK-20603: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20603: Assignee: Apache Spark (was: Shixiong Zhu) > Flaky test:

[jira] [Assigned] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20603: Assignee: Shixiong Zhu (was: Apache Spark) > Flaky test:

[jira] [Created] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

2017-05-04 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-20603: Summary: Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0 Key: SPARK-20603 URL:

[jira] [Commented] (SPARK-20602) Adding LBFGS as optimizer for LinearSVC

2017-05-04 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997314#comment-15997314 ] yuhao yang commented on SPARK-20602: cc [~josephkb] > Adding LBFGS as optimizer for LinearSVC >

[jira] [Assigned] (SPARK-20602) Adding LBFGS as optimizer for LinearSVC

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20602: Assignee: Apache Spark > Adding LBFGS as optimizer for LinearSVC >

[jira] [Assigned] (SPARK-20602) Adding LBFGS as optimizer for LinearSVC

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20602: Assignee: (was: Apache Spark) > Adding LBFGS as optimizer for LinearSVC >

[jira] [Commented] (SPARK-20602) Adding LBFGS as optimizer for LinearSVC

2017-05-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997300#comment-15997300 ] Apache Spark commented on SPARK-20602: -- User 'hhbyyh' has created a pull request for this issue:

[jira] [Created] (SPARK-20602) Adding LBFGS as optimizer for LinearSVC

2017-05-04 Thread yuhao yang (JIRA)
yuhao yang created SPARK-20602: -- Summary: Adding LBFGS as optimizer for LinearSVC Key: SPARK-20602 URL: https://issues.apache.org/jira/browse/SPARK-20602 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997240#comment-15997240 ] Shixiong Zhu commented on SPARK-20599: -- Good point. Yeah, we can just change it to be a longer

[jira] [Comment Edited] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

2017-05-04 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997233#comment-15997233 ] Jacek Laskowski edited comment on SPARK-20599 at 5/4/17 6:59 PM: - Why

[jira] [Commented] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

2017-05-04 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997233#comment-15997233 ] Jacek Laskowski commented on SPARK-20599: - Why can't it work for batch queries? It just seems a

[jira] [Updated] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20599: - Affects Version/s: (was: 2.3.0) 2.2.0 > KafkaSourceProvider should

[jira] [Commented] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997222#comment-15997222 ] Shixiong Zhu commented on SPARK-20599: -- Looks like we just need to provide a better message.

[jira] [Commented] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

2017-05-04 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997219#comment-15997219 ] Jacek Laskowski commented on SPARK-20600: - Couldn't be happier! Thanks [~zsxwing]! >

[jira] [Updated] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20600: - Affects Version/s: (was: 2.3.0) 2.2.0 > KafkaRelation should be

[jira] [Updated] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20600: - Component/s: (was: SQL) > KafkaRelation should be pretty printed in web UI (Details for

[jira] [Commented] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997217#comment-15997217 ] Shixiong Zhu commented on SPARK-20600: -- Could you submit a PR to fix its "toString" method? >

[jira] [Commented] (SPARK-20563) going to DataFrame to RDD and back changes the schema, if the schema is not explicitly provided

2017-05-04 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997205#comment-15997205 ] Bryan Cutler commented on SPARK-20563: -- I think this is to be expected. An RDD does not define a

[jira] [Created] (SPARK-20601) Python API Changes for Constrained Logistic Regression Params

2017-05-04 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-20601: Summary: Python API Changes for Constrained Logistic Regression Params Key: SPARK-20601 URL: https://issues.apache.org/jira/browse/SPARK-20601 Project: Spark

[jira] [Updated] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

2017-05-04 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-20600: Attachment: kafka-source-scan-webui.png > KafkaRelation should be pretty printed in web UI

[jira] [Created] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

2017-05-04 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20600: --- Summary: KafkaRelation should be pretty printed in web UI (Details for Query) Key: SPARK-20600 URL: https://issues.apache.org/jira/browse/SPARK-20600 Project:

[jira] [Created] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

2017-05-04 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20599: --- Summary: KafkaSourceProvider should work with ConsoleSink Key: SPARK-20599 URL: https://issues.apache.org/jira/browse/SPARK-20599 Project: Spark Issue

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997133#comment-15997133 ] Shixiong Zhu edited comment on SPARK-18057 at 5/4/17 5:50 PM: -- [~helena_e]

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

2017-05-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997133#comment-15997133 ] Shixiong Zhu commented on SPARK-18057: -- [~helena_e] I'm curious why you cannot just update the Kafka

[jira] [Comment Edited] (SPARK-20421) Mark JobProgressListener (and related classes) as deprecated

2017-05-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997091#comment-15997091 ] Marcelo Vanzin edited comment on SPARK-20421 at 5/4/17 5:06 PM: I might

[jira] [Commented] (SPARK-20421) Mark JobProgressListener (and related classes) as deprecated

2017-05-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997091#comment-15997091 ] Marcelo Vanzin commented on SPARK-20421: I might remove it at some point. If you look at my

[jira] [Resolved] (SPARK-20595) Parse the 'SPARK_EXECUTOR_INSTANCES' into the parsed arguments

2017-05-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20595. Resolution: Won't Fix See comment in PR. This is intentional. > Parse the

[jira] [Created] (SPARK-20598) Iterative checkpoints do not get removed from HDFS

2017-05-04 Thread Guillem Palou (JIRA)
Guillem Palou created SPARK-20598: - Summary: Iterative checkpoints do not get removed from HDFS Key: SPARK-20598 URL: https://issues.apache.org/jira/browse/SPARK-20598 Project: Spark Issue

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996936#comment-15996936 ] Hyukjin Kwon commented on SPARK-12467: -- Yea, I do agree with the advantage and the others of your

[jira] [Comment Edited] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996936#comment-15996936 ] Hyukjin Kwon edited comment on SPARK-12467 at 5/4/17 3:37 PM: -- Yea, I do

[jira] [Resolved] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12467. -- Resolution: Won't Fix > Get rid of sorting in Row's constructor in pyspark >

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996901#comment-15996901 ] Maciej Szymkiewicz commented on SPARK-12467: [~hyukjin.kwon] Personally I like {{namedtuple}}

[jira] [Commented] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996799#comment-15996799 ] Sean Owen commented on SPARK-20588: --- That's a good point. It means making many more copies of the

[jira] [Commented] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-04 Thread Ameen Tayyebi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996794#comment-15996794 ] Ameen Tayyebi commented on SPARK-20588: --- You could consider caching per thread as well which would

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996761#comment-15996761 ] Hyukjin Kwon commented on SPARK-12467: -- I added 2.2.0 as I tested this in other JIRAs for testing

[jira] [Updated] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12467: - Affects Version/s: 2.2.0 > Get rid of sorting in Row's constructor in pyspark >

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996758#comment-15996758 ] Hyukjin Kwon commented on SPARK-12467: -- I actually quite like {{**kwargs}} usage and I think

[jira] [Comment Edited] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996702#comment-15996702 ] Maciej Szymkiewicz edited comment on SPARK-12467 at 5/4/17 1:13 PM:

[jira] [Commented] (SPARK-12467) Get rid of sorting in Row's constructor in pyspark

2017-05-04 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996702#comment-15996702 ] Maciej Szymkiewicz commented on SPARK-12467: ??Row has named fields, so it shouldn't depend

[jira] [Assigned] (SPARK-20566) ColumnVector should support `appendFloats` for array

2017-05-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20566: --- Assignee: Dongjoon Hyun > ColumnVector should support `appendFloats` for array >

[jira] [Resolved] (SPARK-20566) ColumnVector should support `appendFloats` for array

2017-05-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20566. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 17836

[jira] [Commented] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Jinhua Fu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996681#comment-15996681 ] Jinhua Fu commented on SPARK-20591: --- Does it need modify and may I take this PR? > Succeeded tasks num

[jira] [Commented] (SPARK-20503) ML 2.2 QA: API: Python API coverage

2017-05-04 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996676#comment-15996676 ] Nick Pentreath commented on SPARK-20503: cc [~holdenk] [~bryanc] [~zero323]? I can take it if

[jira] [Commented] (SPARK-20501) ML, Graph 2.2 QA: API: New Scala APIs, docs

2017-05-04 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996675#comment-15996675 ] Nick Pentreath commented on SPARK-20501: Things that would need to be checked include: #

[jira] [Updated] (SPARK-20499) Spark MLlib, GraphX 2.2 QA umbrella

2017-05-04 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-20499: --- Description: This JIRA lists tasks for the next Spark release's QA period for MLlib and

[jira] [Updated] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

2017-05-04 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-20597: Description: # {{KafkaSourceProvider}} supports {{topic}} option that sets the Kafka topic

[jira] [Updated] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

2017-05-04 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-20597: Description: 1. {{KafkaSourceProvider}} supports {{topic}} option that sets the Kafka

[jira] [Created] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

2017-05-04 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20597: --- Summary: KafkaSourceProvider falls back on path as synonym for topic Key: SPARK-20597 URL: https://issues.apache.org/jira/browse/SPARK-20597 Project: Spark

[jira] [Comment Edited] (SPARK-20570) The main version number on docs/latest/index.html

2017-05-04 Thread liucht-inspur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996570#comment-15996570 ] liucht-inspur edited comment on SPARK-20570 at 5/4/17 11:24 AM: Version

[jira] [Closed] (SPARK-20570) The main version number on docs/latest/index.html

2017-05-04 Thread liucht-inspur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liucht-inspur closed SPARK-20570. - Version problem solved and looks like good running > The main version number on

[jira] [Commented] (SPARK-20144) spark.read.parquet no long maintains ordering of the data

2017-05-04 Thread Bill (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996517#comment-15996517 ] Bill commented on SPARK-20144: -- Increasing {{spark.sql.files.openCostInBytes}} prevents the individual

[jira] [Assigned] (SPARK-20574) Allow Bucketizer to handle non-Double column

2017-05-04 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-20574: --- Shepherd: Yanbo Liang Assignee: Wayne Zhang > Allow Bucketizer to handle non-Double

[jira] [Commented] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996475#comment-15996475 ] Sean Owen commented on SPARK-20588: --- It does cache, but the method itself is synchronized on the

[jira] [Updated] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Jinhua Fu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinhua Fu updated SPARK-20591: -- Description: when spark.speculation is enabled,and there are some speculative tasks, then we can see

[jira] [Commented] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996445#comment-15996445 ] Sean Owen commented on SPARK-20591: --- I agree, it's not consistent. I'm not sure what the nature of the

[jira] [Updated] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Jinhua Fu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinhua Fu updated SPARK-20591: -- Attachment: job detail page(stages).png > Succeeded tasks num not equal in job page and job detail

[jira] [Updated] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Jinhua Fu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinhua Fu updated SPARK-20591: -- Attachment: (was: screenshot-1.png) > Succeeded tasks num not equal in job page and job detail

[jira] [Updated] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Jinhua Fu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinhua Fu updated SPARK-20591: -- Attachment: screenshot-1.png > Succeeded tasks num not equal in job page and job detail page on spark

[jira] [Updated] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Jinhua Fu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinhua Fu updated SPARK-20591: -- Attachment: job page.png > Succeeded tasks num not equal in job page and job detail page on spark web

[jira] [Updated] (SPARK-20591) Succeeded tasks num not equal in job page and job detail page on spark web ui when speculative task(s) exist

2017-05-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20591: -- Priority: Minor (was: Major) Issue Type: Bug (was: Improvement) Provide an example please? at

[jira] [Commented] (SPARK-20571) Flaky SparkR StructuredStreaming tests

2017-05-04 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996390#comment-15996390 ] Felix Cheung commented on SPARK-20571: -- done. will monitor tests for a couple of days > Flaky

  1   2   >