[jira] [Created] (SPARK-32116) Python RDD containing a 'pyarrow record_batch object' to java RDD conversion issue

2020-06-27 Thread Tanveer (Jira)
Tanveer created SPARK-32116: --- Summary: Python RDD containing a 'pyarrow record_batch object' to java RDD conversion issue Key: SPARK-32116 URL: https://issues.apache.org/jira/browse/SPARK-32116 Project:

[jira] [Commented] (SPARK-32115) Incorrect results for SUBSTRING when overflow

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147204#comment-17147204 ] Apache Spark commented on SPARK-32115: -- User 'xuanyuanking' has created a pull request for this

[jira] [Assigned] (SPARK-32115) Incorrect results for SUBSTRING when overflow

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32115: Assignee: (was: Apache Spark) > Incorrect results for SUBSTRING when overflow >

[jira] [Assigned] (SPARK-32115) Incorrect results for SUBSTRING when overflow

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32115: Assignee: Apache Spark > Incorrect results for SUBSTRING when overflow >

[jira] [Updated] (SPARK-32115) Incorrect results for SUBSTRING when overflow

2020-06-27 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-32115: Description: SQL query SELECT SUBSTRING("abc", -1207959552, -1207959552) incorrectly returns

[jira] [Updated] (SPARK-32112) Easier way to repartition/coalesce DataFrames based on the number of parallel tasks that Spark can process at the same time

2020-06-27 Thread Noritaka Sekiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-32112: -- Description: Repartition/coalesce is very important to optimize Spark application's

[jira] [Created] (SPARK-32115) Incorrect results for SUBSTRING when overflow

2020-06-27 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-32115: --- Summary: Incorrect results for SUBSTRING when overflow Key: SPARK-32115 URL: https://issues.apache.org/jira/browse/SPARK-32115 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-32112) Easier way to repartition/coalesce DataFrames based on the number of parallel tasks that Spark can process at the same time

2020-06-27 Thread Noritaka Sekiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-32112: -- Summary: Easier way to repartition/coalesce DataFrames based on the number of

[jira] [Commented] (SPARK-30798) Scope Session.active in QueryExecution

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147189#comment-17147189 ] Apache Spark commented on SPARK-30798: -- User 'xuanyuanking' has created a pull request for this

[jira] [Commented] (SPARK-20680) Spark-sql do not support for void column datatype of view

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147152#comment-17147152 ] Apache Spark commented on SPARK-20680: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Commented] (SPARK-20680) Spark-sql do not support for void column datatype of view

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147153#comment-17147153 ] Apache Spark commented on SPARK-20680: -- User 'LantaoJin' has created a pull request for this issue:

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2020-06-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147144#comment-17147144 ] Dongjoon Hyun commented on SPARK-25075: --- It's great! Thanks, [~smarter]. > Build and test Spark

[jira] [Assigned] (SPARK-32071) Benchmark make_interval

2020-06-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32071: - Assignee: Maxim Gekk > Benchmark make_interval > --- > >

[jira] [Resolved] (SPARK-32071) Benchmark make_interval

2020-06-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32071. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28905

[jira] [Commented] (SPARK-32114) Change name of the slaves file, to something more acceptable

2020-06-27 Thread Arvind Krishnan Iyer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147129#comment-17147129 ] Arvind Krishnan Iyer commented on SPARK-32114: -- A suggestion : the file could be named

[jira] [Created] (SPARK-32114) Change name of the slaves file, to something more acceptable

2020-06-27 Thread Arvind Krishnan Iyer (Jira)
Arvind Krishnan Iyer created SPARK-32114: Summary: Change name of the slaves file, to something more acceptable Key: SPARK-32114 URL: https://issues.apache.org/jira/browse/SPARK-32114

[jira] [Commented] (SPARK-31823) Improve the current Spark Scheduler test framework

2020-06-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147034#comment-17147034 ] Dongjoon Hyun commented on SPARK-31823: --- Hi, [~beliefer]. What do you mean? Could you give us some

[jira] [Updated] (SPARK-31823) Improve the current Spark Scheduler test framework

2020-06-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31823: -- Priority: Minor (was: Major) > Improve the current Spark Scheduler test framework >

[jira] [Updated] (SPARK-31847) DAGSchedulerSuite: Rewrite the test framework to cover most of the existing major features of the Spark Scheduler, mock the necessary part wisely, and make the test fram

2020-06-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31847: -- Component/s: Tests > DAGSchedulerSuite: Rewrite the test framework to cover most of the

[jira] [Updated] (SPARK-31823) Improve the current Spark Scheduler test framework

2020-06-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31823: -- Component/s: Tests > Improve the current Spark Scheduler test framework >

[jira] [Commented] (SPARK-32113) Avoid coalescing shuffle partitions if joining event-based table

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147025#comment-17147025 ] Apache Spark commented on SPARK-32113: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-32113) Avoid coalescing shuffle partitions if joining event-based table

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147023#comment-17147023 ] Apache Spark commented on SPARK-32113: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32113) Avoid coalescing shuffle partitions if joining event-based table

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32113: Assignee: (was: Apache Spark) > Avoid coalescing shuffle partitions if joining

[jira] [Assigned] (SPARK-32113) Avoid coalescing shuffle partitions if joining event-based table

2020-06-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32113: Assignee: Apache Spark > Avoid coalescing shuffle partitions if joining event-based

[jira] [Updated] (SPARK-32113) Avoid coalescing shuffle partitions if joining event-based table

2020-06-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32113: Attachment: disable.png default.png > Avoid coalescing shuffle partitions if

[jira] [Created] (SPARK-32113) Avoid coalescing shuffle partitions if joining event-based table

2020-06-27 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-32113: --- Summary: Avoid coalescing shuffle partitions if joining event-based table Key: SPARK-32113 URL: https://issues.apache.org/jira/browse/SPARK-32113 Project: Spark

[jira] [Commented] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-27 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147012#comment-17147012 ] Frank Oosterhuis commented on SPARK-32051: -- Okay, thanks :) > Dataset.foreachPartition returns

[jira] [Commented] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146950#comment-17146950 ] Sean R. Owen commented on SPARK-32051: -- Oh, that's quite different. You can't call Spark within

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2020-06-27 Thread Guillaume Martres (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146941#comment-17146941 ] Guillaume Martres commented on SPARK-25075: --- 2.13.3 has just been released with a fix for the

[jira] [Commented] (SPARK-32051) Dataset.foreachPartition returns object

2020-06-27 Thread Frank Oosterhuis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146866#comment-17146866 ] Frank Oosterhuis commented on SPARK-32051: -- How would you work around the .foreach situation?

[jira] [Commented] (SPARK-32104) Avoid full outer join OOM on skewed dataset

2020-06-27 Thread Zuo Dao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146839#comment-17146839 ] Zuo Dao commented on SPARK-32104: - Yes, same problem. [~viirya] > Avoid full outer join OOM on skewed

[jira] [Commented] (SPARK-32104) Avoid full outer join OOM on skewed dataset

2020-06-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146806#comment-17146806 ] L. C. Hsieh commented on SPARK-32104: - Is this duplicate to SPARK-24985? > Avoid full outer join