[jira] [Assigned] (SPARK-23245) KafkaContinuousSourceSuite may hang forever

2018-01-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-23245: Assignee: Jose Torres > KafkaContinuousSourceSuite may hang forever >

[jira] [Resolved] (SPARK-23245) KafkaContinuousSourceSuite may hang forever

2018-01-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-23245. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20413

[jira] [Assigned] (SPARK-23247) combines Unsafe operations and statistics operations in Scan Data Source

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23247: Assignee: Apache Spark > combines Unsafe operations and statistics operations in Scan

[jira] [Assigned] (SPARK-23247) combines Unsafe operations and statistics operations in Scan Data Source

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23247: Assignee: (was: Apache Spark) > combines Unsafe operations and statistics operations

[jira] [Commented] (SPARK-23247) combines Unsafe operations and statistics operations in Scan Data Source

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341996#comment-16341996 ] Apache Spark commented on SPARK-23247: -- User 'heary-cao' has created a pull request for this issue:

[jira] [Created] (SPARK-23247) combines Unsafe operations and statistics operations in Scan Data Source

2018-01-26 Thread caoxuewen (JIRA)
caoxuewen created SPARK-23247: - Summary: combines Unsafe operations and statistics operations in Scan Data Source Key: SPARK-23247 URL: https://issues.apache.org/jira/browse/SPARK-23247 Project: Spark

[jira] [Updated] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-26 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MBA Learns to Code updated SPARK-23246: --- Summary: (Py)Spark OOM because of iteratively accumulated metadata that cannot be

[jira] [Updated] (SPARK-23246) (Py)Spark OOM because of metadata build-up that cannot be cleaned

2018-01-26 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MBA Learns to Code updated SPARK-23246: --- Description: I am having consistent OOM crashes when trying to use PySpark for

[jira] [Updated] (SPARK-23246) (Py)Spark OOM because of metadata build-up that cannot be cleaned

2018-01-26 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MBA Learns to Code updated SPARK-23246: --- Description: I am having consistent OOM crashes when trying to use PySpark for

[jira] [Created] (SPARK-23246) (Py)Spark OOM because of metadata build-up that cannot be cleaned

2018-01-26 Thread MBA Learns to Code (JIRA)
MBA Learns to Code created SPARK-23246: -- Summary: (Py)Spark OOM because of metadata build-up that cannot be cleaned Key: SPARK-23246 URL: https://issues.apache.org/jira/browse/SPARK-23246

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

2018-01-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341889#comment-16341889 ] Wenchen Fan commented on SPARK-16441: - I believe this issue has been fixed in Spark 2.3 by 

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

2018-01-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341888#comment-16341888 ] Wenchen Fan commented on SPARK-21460: - I believe this issue has been fixed in Spark 2.3 by 

[jira] [Comment Edited] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341877#comment-16341877 ] Weichen Xu edited comment on SPARK-23110 at 1/27/18 1:19 AM: - I use the

[jira] [Commented] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341877#comment-16341877 ] Weichen Xu commented on SPARK-23110: I use the script `1_process_script` to generate some info (see

[jira] [Updated] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-23110: --- Attachment: 1_process_script.sh > ML 2.3 QA: API: Java compatibility, docs >

[jira] [Updated] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-23110: --- Attachment: added_ml_class > ML 2.3 QA: API: Java compatibility, docs >

[jira] [Updated] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-23110: --- Attachment: different_methods_in_ML.diff > ML 2.3 QA: API: Java compatibility, docs >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-26 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: ExecutorsTab.png) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-26 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: ExecutorsTab.png > Additional Memory Tuning Metrics > > >

[jira] [Resolved] (SPARK-23214) cached data should not carry extra hint info

2018-01-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23214. - Resolution: Fixed Fix Version/s: 2.3.0 > cached data should not carry extra hint info >

[jira] [Commented] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341859#comment-16341859 ] Apache Spark commented on SPARK-23243: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23243: Assignee: Apache Spark > Shuffle+Repartition on an RDD could lead to incorrect answers >

[jira] [Assigned] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23243: Assignee: (was: Apache Spark) > Shuffle+Repartition on an RDD could lead to incorrect

[jira] [Assigned] (SPARK-23245) KafkaContinuousSourceSuite may hang forever

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23245: Assignee: (was: Apache Spark) > KafkaContinuousSourceSuite may hang forever >

[jira] [Commented] (SPARK-23245) KafkaContinuousSourceSuite may hang forever

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341846#comment-16341846 ] Apache Spark commented on SPARK-23245: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23245) KafkaContinuousSourceSuite may hang forever

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23245: Assignee: Apache Spark > KafkaContinuousSourceSuite may hang forever >

[jira] [Created] (SPARK-23245) KafkaContinuousSourceSuite may hang forever

2018-01-26 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-23245: Summary: KafkaContinuousSourceSuite may hang forever Key: SPARK-23245 URL: https://issues.apache.org/jira/browse/SPARK-23245 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-23242. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20412

[jira] [Reopened] (SPARK-22797) Add multiple column support to PySpark Bucketizer

2018-01-26 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal reopened SPARK-22797: > Add multiple column support to PySpark Bucketizer >

[jira] [Updated] (SPARK-22797) Add multiple column support to PySpark Bucketizer

2018-01-26 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-22797: --- Fix Version/s: (was: 2.3.0) > Add multiple column support to PySpark Bucketizer >

[jira] [Commented] (SPARK-23220) broadcast hint not applied in a streaming left anti join

2018-01-26 Thread Mathieu DESPRIEE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341791#comment-16341791 ] Mathieu DESPRIEE commented on SPARK-23220: -- Here is a gist

[jira] [Issue Comment Deleted] (SPARK-23220) broadcast hint not applied in a streaming left anti join

2018-01-26 Thread Mathieu DESPRIEE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu DESPRIEE updated SPARK-23220: - Comment: was deleted (was: [~viirya] working on it. It's actually harder than I thought.

[jira] [Updated] (SPARK-23244) Incorrect handling of default values when deserializing python wrappers of scala transformers

2018-01-26 Thread Tomas Nykodym (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomas Nykodym updated SPARK-23244: -- Description: Default values are not handled properly when serializing/deserializing python

[jira] [Created] (SPARK-23244) Incorrect handling of default values when deserializing python wrappers of scala transformers

2018-01-26 Thread Tomas Nykodym (JIRA)
Tomas Nykodym created SPARK-23244: - Summary: Incorrect handling of default values when deserializing python wrappers of scala transformers Key: SPARK-23244 URL: https://issues.apache.org/jira/browse/SPARK-23244

[jira] [Resolved] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-01-26 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal resolved SPARK-23207. Resolution: Fixed Fix Version/s: 2.4.0 2.3.0 Issue resolved by

[jira] [Created] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-01-26 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-23243: Summary: Shuffle+Repartition on an RDD could lead to incorrect answers Key: SPARK-23243 URL: https://issues.apache.org/jira/browse/SPARK-23243 Project: Spark

[jira] [Updated] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-01-26 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-23207: - Summary: Shuffle+Repartition on an DataFrame could lead to incorrect answers (was:

[jira] [Commented] (SPARK-23237) Add UI / endpoint for threaddumps for executors with active tasks

2018-01-26 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341730#comment-16341730 ] Alex Bozarth commented on SPARK-23237: -- Unlike the related task, I'm not sure about this one. I see

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-01-26 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341727#comment-16341727 ] Alex Bozarth commented on SPARK-23235: -- I think I would be ok with adding threadDump to the api, but

[jira] [Commented] (SPARK-23236) Make it easier to find the rest API, especially in local mode

2018-01-26 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341724#comment-16341724 ] Alex Bozarth commented on SPARK-23236: -- So IIUC, you want 1. /api and /api/v1 to give the same

[jira] [Assigned] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23242: Assignee: Shixiong Zhu (was: Apache Spark) > Don't run tests in KafkaSourceSuiteBase

[jira] [Commented] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341680#comment-16341680 ] Apache Spark commented on SPARK-23242: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23242: Assignee: Apache Spark (was: Shixiong Zhu) > Don't run tests in KafkaSourceSuiteBase

[jira] [Updated] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-23242: - Description: KafkaSourceSuiteBase should be abstract class, otherwise KafkaSourceSuiteBase will

[jira] [Updated] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-23242: - Component/s: Tests > Don't run tests in KafkaSourceSuiteBase twice >

[jira] [Updated] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-23242: - Environment: (was: KafkaSourceSuiteBase should be abstract class, otherwise

[jira] [Created] (SPARK-23242) Don't run tests in KafkaSourceSuiteBase twice

2018-01-26 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-23242: Summary: Don't run tests in KafkaSourceSuiteBase twice Key: SPARK-23242 URL: https://issues.apache.org/jira/browse/SPARK-23242 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-23241) from_unixtime SQL function returning incorrect dates

2018-01-26 Thread Luke R Hospadaruk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke R Hospadaruk resolved SPARK-23241. --- Resolution: Not A Problem Bad date format strings - see

[jira] [Commented] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-26 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341609#comment-16341609 ] Bruce Robbins commented on SPARK-23240: --- I will be making a pull request. > PythonWorkerFactory

[jira] [Commented] (SPARK-23241) from_unixtime SQL function returning incorrect dates

2018-01-26 Thread Luke R Hospadaruk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341593#comment-16341593 ] Luke R Hospadaruk commented on SPARK-23241: --- Not sure if I prioritized this correctly? >

[jira] [Created] (SPARK-23241) from_unixtime SQL function returning incorrect dates

2018-01-26 Thread Luke R Hospadaruk (JIRA)
Luke R Hospadaruk created SPARK-23241: - Summary: from_unixtime SQL function returning incorrect dates Key: SPARK-23241 URL: https://issues.apache.org/jira/browse/SPARK-23241 Project: Spark

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-26 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341526#comment-16341526 ] Edwina Lu commented on SPARK-23206: --- We'd like to monitor the following executor level metrics: * JVM

[jira] [Created] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-26 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23240: - Summary: PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout Key: SPARK-23240 URL: https://issues.apache.org/jira/browse/SPARK-23240

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-26 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: ExecutorsTab.png > Additional Memory Tuning Metrics > > >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-26 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: ExecutorsTab.png) > Additional Memory Tuning Metrics >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-26 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: ExecutorsTab.png > Additional Memory Tuning Metrics > > >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-26 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: StageTab.png > Additional Memory Tuning Metrics > > >

[jira] [Commented] (SPARK-17139) Add model summary for MultinomialLogisticRegression

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341434#comment-16341434 ] Apache Spark commented on SPARK-17139: -- User 'WeichenXu123' has created a pull request for this

[jira] [Commented] (SPARK-17139) Add model summary for MultinomialLogisticRegression

2018-01-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341420#comment-16341420 ] Weichen Xu commented on SPARK-17139: [~mlnick] Yes it breaks binary compatibility. But we found no

[jira] [Updated] (SPARK-23239) KafkaRelationSuite should clean up its continuous queries

2018-01-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23239: -- Description: Currently, `KafkaRelationSuite` doesn't clean up its continuous microbatch

[jira] [Updated] (SPARK-23239) KafkaRelationSuite should clean up its continuous queries

2018-01-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23239: -- Description: Currently, `KafkaRelationSuite` doesn't clean up its continuous queries. As a

[jira] [Created] (SPARK-23239) KafkaRelationSuite should clean up its continuous queries

2018-01-26 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-23239: - Summary: KafkaRelationSuite should clean up its continuous queries Key: SPARK-23239 URL: https://issues.apache.org/jira/browse/SPARK-23239 Project: Spark

[jira] [Comment Edited] (SPARK-23213) SparkR:::textFile(sc1,"/opt/test333") can not work on spark2.2.1

2018-01-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341316#comment-16341316 ] Hyukjin Kwon edited comment on SPARK-23213 at 1/26/18 5:25 PM: --- Google what

[jira] [Commented] (SPARK-23213) SparkR:::textFile(sc1,"/opt/test333") can not work on spark2.2.1

2018-01-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341316#comment-16341316 ] Hyukjin Kwon commented on SPARK-23213: -- Google what triple columns in R mean ... Questions should go

[jira] [Commented] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341315#comment-16341315 ] Apache Spark commented on SPARK-23238: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23238: Assignee: Apache Spark > Externalize SQLConf spark.sql.execution.arrow.enabled >

[jira] [Assigned] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23238: Assignee: (was: Apache Spark) > Externalize SQLConf spark.sql.execution.arrow.enabled

***UNCHECKED*** [jira] [Resolved] (SPARK-23218) simplify ColumnVector.getArray

2018-01-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23218. - Resolution: Fixed Fix Version/s: 2.3.0 > simplify ColumnVector.getArray >

[jira] [Commented] (SPARK-23213) SparkR:::textFile(sc1,"/opt/test333") can not work on spark2.2.1

2018-01-26 Thread Tony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341286#comment-16341286 ] Tony commented on SPARK-23213: --- Hi, [~felixcheung] Another question here, what's the mechanism to make a

[jira] [Updated] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23238: Issue Type: Improvement (was: Bug) > Externalize SQLConf spark.sql.execution.arrow.enabled >

[jira] [Updated] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23238: Target Version/s: 2.3.0 > Externalize SQLConf spark.sql.execution.arrow.enabled >

[jira] [Created] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-26 Thread Xiao Li (JIRA)
Xiao Li created SPARK-23238: --- Summary: Externalize SQLConf spark.sql.execution.arrow.enabled Key: SPARK-23238 URL: https://issues.apache.org/jira/browse/SPARK-23238 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-26 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-23189: --- Attachment: (was: multiple_stages_2.png) > reflect stage level blacklisting on

[jira] [Updated] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-26 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-23189: --- Attachment: multiple_stages_3.png > reflect stage level blacklisting on executor tab

[jira] [Updated] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-26 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-23189: --- Attachment: (was: multiple_stages_3.png) > reflect stage level blacklisting on

[jira] [Updated] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-26 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-23189: --- Attachment: multiple_stages_2.png > reflect stage level blacklisting on executor tab

[jira] [Commented] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-26 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341232#comment-16341232 ] Felix Cheung commented on SPARK-23107: -- Thanks My bad RFormula does have a page

[jira] [Created] (SPARK-23237) Add UI / endpoint for threaddumps for executors with active tasks

2018-01-26 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23237: Summary: Add UI / endpoint for threaddumps for executors with active tasks Key: SPARK-23237 URL: https://issues.apache.org/jira/browse/SPARK-23237 Project: Spark

[jira] [Commented] (SPARK-17139) Add model summary for MultinomialLogisticRegression

2018-01-26 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341213#comment-16341213 ] Seth Hendrickson commented on SPARK-17139: -- Good catch, apart from redesigning this patch, I'm

[jira] [Created] (SPARK-23236) Make it easier to find the rest API, especially in local mode

2018-01-26 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23236: Summary: Make it easier to find the rest API, especially in local mode Key: SPARK-23236 URL: https://issues.apache.org/jira/browse/SPARK-23236 Project: Spark

[jira] [Updated] (SPARK-23236) Make it easier to find the rest API, especially in local mode

2018-01-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23236: - Labels: newbie (was: ) > Make it easier to find the rest API, especially in local mode >

[jira] [Assigned] (SPARK-23234) ML python test failure due to default outputCol

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23234: Assignee: Apache Spark > ML python test failure due to default outputCol >

[jira] [Commented] (SPARK-23234) ML python test failure due to default outputCol

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341211#comment-16341211 ] Apache Spark commented on SPARK-23234: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23234) ML python test failure due to default outputCol

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23234: Assignee: (was: Apache Spark) > ML python test failure due to default outputCol >

[jira] [Updated] (SPARK-23234) ML python test failure due to default outputCol

2018-01-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-23234: Description: SPARK-22799 and SPARK-22797 are causing valid Python test failures. The reason is

[jira] [Updated] (SPARK-23234) ML python test failure due to default outputCol

2018-01-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-23234: Summary: ML python test failure due to default outputCol (was: ML python test failure) > ML

[jira] [Updated] (SPARK-23234) ML python test failure

2018-01-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-23234: Description: SPARK-22799 and SPARK-22797 are causing valid Python test failures. The reason is

[jira] [Updated] (SPARK-23235) Add executor Threaddump to api

2018-01-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23235: - Priority: Minor (was: Major) > Add executor Threaddump to api > --

[jira] [Created] (SPARK-23235) Add executor Threaddump to api

2018-01-26 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-23235: Summary: Add executor Threaddump to api Key: SPARK-23235 URL: https://issues.apache.org/jira/browse/SPARK-23235 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-23235) Add executor Threaddump to api

2018-01-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23235: - Labels: newbie (was: ) > Add executor Threaddump to api > -- > >

[jira] [Created] (SPARK-23234) ML python test failure

2018-01-26 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-23234: --- Summary: ML python test failure Key: SPARK-23234 URL: https://issues.apache.org/jira/browse/SPARK-23234 Project: Spark Issue Type: Bug Components:

***UNCHECKED*** [jira] [Assigned] (SPARK-23233) asNondeterministic in Python UDF not being set when the UDF is called at least once

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23233: Assignee: (was: Apache Spark) > asNondeterministic in Python UDF not being set when

[jira] [Assigned] (SPARK-23233) asNondeterministic in Python UDF not being set when the UDF is called at least once

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23233: Assignee: Apache Spark > asNondeterministic in Python UDF not being set when the UDF is

[jira] [Commented] (SPARK-23233) asNondeterministic in Python UDF not being set when the UDF is called at least once

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341143#comment-16341143 ] Apache Spark commented on SPARK-23233: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-23220) broadcast hint not applied in a streaming left anti join

2018-01-26 Thread Mathieu DESPRIEE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341142#comment-16341142 ] Mathieu DESPRIEE commented on SPARK-23220: -- [~viirya] working on it. It's actually harder than I

***UNCHECKED*** [jira] [Created] (SPARK-23233) asNondeterministic in Python UDF not being set when the UDF is called at least once

2018-01-26 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-23233: Summary: asNondeterministic in Python UDF not being set when the UDF is called at least once Key: SPARK-23233 URL: https://issues.apache.org/jira/browse/SPARK-23233

[jira] [Assigned] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23189: Assignee: (was: Apache Spark) > reflect stage level blacklisting on executor tab >

[jira] [Assigned] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23189: Assignee: Apache Spark > reflect stage level blacklisting on executor tab >

[jira] [Commented] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-01-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341139#comment-16341139 ] Apache Spark commented on SPARK-23189: -- User 'attilapiros' has created a pull request for this

[jira] [Updated] (SPARK-23232) Mapping Dataset to a Java bean always set 1L to a long field

2018-01-26 Thread Hristo Angelov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hristo Angelov updated SPARK-23232: --- Description: I have the following streaming query:  {code:java} baseDataSet

  1   2   >