[jira] [Commented] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Michael (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149922#comment-17149922 ] Michael commented on SPARK-32148: - [~cloud_fan], yes the issue is with stream-stream joins. Stream-batch

[jira] [Commented] (SPARK-31693) Investigate AmpLab Jenkins server network issue

2020-07-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149874#comment-17149874 ] Hyukjin Kwon commented on SPARK-31693: -- [~shaneknapp] .. seems this popped up again ;(. >

[jira] [Reopened] (SPARK-31693) Investigate AmpLab Jenkins server network issue

2020-07-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-31693: -- > Investigate AmpLab Jenkins server network issue >

[jira] [Resolved] (SPARK-32136) Spark producing incorrect groupBy results when key is a struct with nullable properties

2020-07-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32136. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32136) Spark producing incorrect groupBy results when key is a struct with nullable properties

2020-07-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32136: Assignee: L. C. Hsieh > Spark producing incorrect groupBy results when key is a struct

[jira] [Commented] (SPARK-31976) use MemoryUsage to control the size of block

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149842#comment-17149842 ] Apache Spark commented on SPARK-31976: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-31976) use MemoryUsage to control the size of block

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31976: Assignee: (was: Apache Spark) > use MemoryUsage to control the size of block >

[jira] [Commented] (SPARK-31976) use MemoryUsage to control the size of block

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149841#comment-17149841 ] Apache Spark commented on SPARK-31976: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-31976) use MemoryUsage to control the size of block

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31976: Assignee: Apache Spark > use MemoryUsage to control the size of block >

[jira] [Commented] (SPARK-31935) Hadoop file system config should be effective in data source options

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149799#comment-17149799 ] Apache Spark commented on SPARK-31935: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-31935) Hadoop file system config should be effective in data source options

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149798#comment-17149798 ] Apache Spark commented on SPARK-31935: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-32151) Kafka does not allow Partition Rebalance Handling

2020-07-01 Thread Ed Mitchell (Jira)
Ed Mitchell created SPARK-32151: --- Summary: Kafka does not allow Partition Rebalance Handling Key: SPARK-32151 URL: https://issues.apache.org/jira/browse/SPARK-32151 Project: Spark Issue Type:

[jira] [Updated] (SPARK-32151) Kafka does not allow Partition Rebalance Handling

2020-07-01 Thread Ed Mitchell (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ed Mitchell updated SPARK-32151: Description: When a consumer group rebalance occurs when the Spark driver is using the Subscribe

[jira] [Updated] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-32148: Priority: Blocker (was: Major) > LEFT JOIN generating non-deterministic and unexpected result

[jira] [Commented] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149780#comment-17149780 ] Wenchen Fan commented on SPARK-32148: - I assume this is for streaming join? I'm marking it as a

[jira] [Resolved] (SPARK-29465) Unable to configure SPARK UI (spark.ui.port) in spark yarn cluster mode.

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29465. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28880

[jira] [Commented] (SPARK-30794) Stage Level scheduling: Add ability to set off heap memory

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149759#comment-17149759 ] Apache Spark commented on SPARK-30794: -- User 'warrenzhu25' has created a pull request for this

[jira] [Assigned] (SPARK-30794) Stage Level scheduling: Add ability to set off heap memory

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30794: Assignee: (was: Apache Spark) > Stage Level scheduling: Add ability to set off heap

[jira] [Assigned] (SPARK-30794) Stage Level scheduling: Add ability to set off heap memory

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30794: Assignee: Apache Spark > Stage Level scheduling: Add ability to set off heap memory >

[jira] [Commented] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149754#comment-17149754 ] Jungtaek Lim commented on SPARK-32148: -- Looking into it. Looks like a problem indeed (if then this

[jira] [Assigned] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32130: - Assignee: Maxim Gekk > Spark 3.0 json load performance is unacceptable in comparison

[jira] [Resolved] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32130. --- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by

[jira] [Updated] (SPARK-32026) Add PrometheusServletSuite

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32026: -- Issue Type: Improvement (was: Task) > Add PrometheusServletSuite >

[jira] [Updated] (SPARK-32026) Add PrometheusServletSuite

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32026: -- Priority: Minor (was: Major) > Add PrometheusServletSuite > -- > >

[jira] [Updated] (SPARK-32026) Add PrometheusServletSuite

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32026: -- Component/s: Tests > Add PrometheusServletSuite > -- > >

[jira] [Updated] (SPARK-32026) Add PrometheusServletSuite

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32026: -- Affects Version/s: (was: 3.0.1) 3.1.0 > Add PrometheusServletSuite

[jira] [Resolved] (SPARK-32026) Add PrometheusServletSuite

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32026. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28865

[jira] [Assigned] (SPARK-32026) Add PrometheusServletSuite

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32026: - Assignee: Eren Avsarogullari > Add PrometheusServletSuite > --

[jira] [Assigned] (SPARK-30010) Remove deprecated SparkConf.setAll(Traversable)

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30010: Assignee: (was: Apache Spark) > Remove deprecated SparkConf.setAll(Traversable) >

[jira] [Commented] (SPARK-29292) Fix internal usages of mutable collection for Seq in 2.13

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149599#comment-17149599 ] Apache Spark commented on SPARK-29292: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-30010) Remove deprecated SparkConf.setAll(Traversable)

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30010: Assignee: Apache Spark > Remove deprecated SparkConf.setAll(Traversable) >

[jira] [Commented] (SPARK-30010) Remove deprecated SparkConf.setAll(Traversable)

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149598#comment-17149598 ] Apache Spark commented on SPARK-30010: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-29292) Fix internal usages of mutable collection for Seq in 2.13

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29292: Assignee: Sean R. Owen (was: Apache Spark) > Fix internal usages of mutable collection

[jira] [Assigned] (SPARK-29292) Fix internal usages of mutable collection for Seq in 2.13

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-29292: Assignee: Apache Spark (was: Sean R. Owen) > Fix internal usages of mutable collection

[jira] [Commented] (SPARK-29292) Fix internal usages of mutable collection for Seq in 2.13

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149597#comment-17149597 ] Apache Spark commented on SPARK-29292: -- User 'srowen' has created a pull request for this issue:

[jira] [Resolved] (SPARK-30132) Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses

2020-07-01 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30132. -- Resolution: Not A Problem I confirmed that Scala 2.13.3 resolves this; no further action

[jira] [Assigned] (SPARK-31723) Flaky test: org.apache.spark.deploy.history.HistoryServerSuite

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31723: Assignee: Apache Spark > Flaky test: org.apache.spark.deploy.history.HistoryServerSuite

[jira] [Assigned] (SPARK-31723) Flaky test: org.apache.spark.deploy.history.HistoryServerSuite

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31723: Assignee: (was: Apache Spark) > Flaky test:

[jira] [Commented] (SPARK-31723) Flaky test: org.apache.spark.deploy.history.HistoryServerSuite

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149574#comment-17149574 ] Apache Spark commented on SPARK-31723: -- User 'warrenzhu25' has created a pull request for this

[jira] [Assigned] (SPARK-32150) Upgrade to Zstd 1.4.5-4

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32150: Assignee: William Hyun (was: Apache Spark) > Upgrade to Zstd 1.4.5-4 >

[jira] [Assigned] (SPARK-32150) Upgrade to Zstd 1.4.5-4

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32150: Assignee: Apache Spark (was: William Hyun) > Upgrade to Zstd 1.4.5-4 >

[jira] [Commented] (SPARK-32150) Upgrade to Zstd 1.4.5-4

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149548#comment-17149548 ] Apache Spark commented on SPARK-32150: -- User 'williamhyun' has created a pull request for this

[jira] [Updated] (SPARK-32150) Upgrade to Zstd 1.4.5-4

2020-07-01 Thread William Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Hyun updated SPARK-32150: - Description: This issue aims to upgrade to Zstd 1.4.5-4 ZStd 1.4.5-4 fixes the following. -

[jira] [Created] (SPARK-32150) Upgrade to Zstd 1.4.5-4

2020-07-01 Thread William Hyun (Jira)
William Hyun created SPARK-32150: Summary: Upgrade to Zstd 1.4.5-4 Key: SPARK-32150 URL: https://issues.apache.org/jira/browse/SPARK-32150 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32130: -- Target Version/s: 3.0.1 > Spark 3.0 json load performance is unacceptable in comparison of

[jira] [Assigned] (SPARK-32149) Improve file path name normalisation at block resolution within the external shuffle service

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32149: Assignee: Apache Spark > Improve file path name normalisation at block resolution within

[jira] [Commented] (SPARK-32149) Improve file path name normalisation at block resolution within the external shuffle service

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149513#comment-17149513 ] Apache Spark commented on SPARK-32149: -- User 'attilapiros' has created a pull request for this

[jira] [Commented] (SPARK-32010) Thread leaks in pinned thread mode

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149512#comment-17149512 ] Apache Spark commented on SPARK-32010: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-32149) Improve file path name normalisation at block resolution within the external shuffle service

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32149: Assignee: (was: Apache Spark) > Improve file path name normalisation at block

[jira] [Commented] (SPARK-32010) Thread leaks in pinned thread mode

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149511#comment-17149511 ] Apache Spark commented on SPARK-32010: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-32010) Thread leaks in pinned thread mode

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32010: Assignee: Hyukjin Kwon (was: Apache Spark) > Thread leaks in pinned thread mode >

[jira] [Assigned] (SPARK-32010) Thread leaks in pinned thread mode

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32010: Assignee: Apache Spark (was: Hyukjin Kwon) > Thread leaks in pinned thread mode >

[jira] [Commented] (SPARK-32121) ExternalShuffleBlockResolverSuite failed on Windows

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149506#comment-17149506 ] Apache Spark commented on SPARK-32121: -- User 'attilapiros' has created a pull request for this

[jira] [Updated] (SPARK-32149) Improve file path name normalisation at block resolution within the external shuffle service

2020-07-01 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-32149: --- Affects Version/s: (was: 3.0.1) 3.1.0 > Improve file

[jira] [Commented] (SPARK-30037) Customize krb5.conf to test Kafka delegation token with MiniKDC

2020-07-01 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149451#comment-17149451 ] Gabor Somogyi commented on SPARK-30037: --- What use-case justifies the need to create a maybe

[jira] [Commented] (SPARK-32149) Improve file path name normalisation at block resolution within the external shuffle service

2020-07-01 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149440#comment-17149440 ] Attila Zsolt Piros commented on SPARK-32149: I am working on this > Improve file path name

[jira] [Created] (SPARK-32149) Improve file path name normalisation at block resolution within the external shuffle service

2020-07-01 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-32149: -- Summary: Improve file path name normalisation at block resolution within the external shuffle service Key: SPARK-32149 URL:

[jira] [Resolved] (SPARK-23631) Add summary to RandomForestClassificationModel

2020-07-01 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-23631. -- Fix Version/s: 3.1.0 Resolution: Fixed Resolved by

[jira] [Updated] (SPARK-23631) Add summary to RandomForestClassificationModel

2020-07-01 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-23631: - Priority: Minor (was: Major) > Add summary to RandomForestClassificationModel >

[jira] [Assigned] (SPARK-23631) Add summary to RandomForestClassificationModel

2020-07-01 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-23631: Assignee: Huaxin Gao > Add summary to RandomForestClassificationModel >

[jira] [Reopened] (SPARK-23631) Add summary to RandomForestClassificationModel

2020-07-01 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reopened SPARK-23631: -- > Add summary to RandomForestClassificationModel > --

[jira] [Updated] (SPARK-32029) Check spark context is stoped when get active session

2020-07-01 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-32029: Summary: Check spark context is stoped when get active session (was: Check active session if

[jira] [Updated] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Michael (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael updated SPARK-32148: Description: When upgrading from Spark 2.4.6 to 3.0.0 I found that previously working LEFT JOINs now

[jira] [Updated] (SPARK-32029) Check active session if spark context is stop

2020-07-01 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-32029: Summary: Check active session if spark context is stop (was: Make activeSession null when

[jira] [Resolved] (SPARK-32147) Spark: PartitionBy changing the columns value

2020-07-01 Thread Shankar Koirala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shankar Koirala resolved SPARK-32147. - Resolution: Not A Problem > Spark: PartitionBy changing the columns value >

[jira] [Commented] (SPARK-32147) Spark: PartitionBy changing the columns value

2020-07-01 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149403#comment-17149403 ] Lantao Jin commented on SPARK-32147: set spark.sql.sources.partitionColumnTypeInference.enabled to

[jira] [Updated] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Michael (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael updated SPARK-32148: Description: When upgrading from Spark 2.4.6 to 3.0.0 I found that previously working LEFT JOINs now

[jira] [Updated] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Michael (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael updated SPARK-32148: Description: When upgrading from Spark 2.4.6 to 3.0.0 I found that previously working LEFT JOINs now

[jira] [Updated] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Michael (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael updated SPARK-32148: Description: When upgrading from Spark 2.4.6 to 3.0.0 I found that previously working LEFT JOINs now

[jira] [Created] (SPARK-32148) LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0)

2020-07-01 Thread Michael (Jira)
Michael created SPARK-32148: --- Summary: LEFT JOIN generating non-deterministic and unexpected result (regression in Spark 3.0) Key: SPARK-32148 URL: https://issues.apache.org/jira/browse/SPARK-32148

[jira] [Assigned] (SPARK-28169) Spark can’t push down partition predicate for OR expression

2020-07-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28169: --- Assignee: angerszhu > Spark can’t push down partition predicate for OR expression >

[jira] [Resolved] (SPARK-28169) Spark can’t push down partition predicate for OR expression

2020-07-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28169. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28805

[jira] [Commented] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149348#comment-17149348 ] Apache Spark commented on SPARK-32130: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149347#comment-17149347 ] Apache Spark commented on SPARK-32130: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32130: Assignee: Apache Spark > Spark 3.0 json load performance is unacceptable in comparison

[jira] [Assigned] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32130: Assignee: (was: Apache Spark) > Spark 3.0 json load performance is unacceptable in

[jira] [Commented] (SPARK-32132) Thriftserver interval returns "4 weeks 2 days" in 2.4 and "30 days" in 3.0

2020-07-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149337#comment-17149337 ] Hyukjin Kwon commented on SPARK-32132: -- Thanks [~juliuszsompolski]. > Thriftserver interval

[jira] [Updated] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-07-01 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-30985: Description: SPARK_CONF_DIR hosts configuration files like, 1) spark-defaults.conf -

[jira] [Updated] (SPARK-30985) Propagate SPARK_CONF_DIR files to driver and exec pods.

2020-07-01 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-30985: Description: SPARK_CONF_DIR hosts configuration files like, 1) spark-defaults.conf -

[jira] [Commented] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149334#comment-17149334 ] Hyukjin Kwon commented on SPARK-32130: -- Yeah, we can disable it back by default considering the

[jira] [Commented] (SPARK-32027) EventLoggingListener threw java.util.ConcurrentModificationException

2020-07-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149321#comment-17149321 ] Hyukjin Kwon commented on SPARK-32027: -- [~yumwang] can you write down more details if you're not

[jira] [Updated] (SPARK-32147) Spark: PartitionBy changing the columns value

2020-07-01 Thread Shankar Koirala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shankar Koirala updated SPARK-32147: Labels: spark (was: ) > Spark: PartitionBy changing the columns value >

[jira] [Commented] (SPARK-31846) DAGSchedulerSuite: For the pattern of cancel + assert, extract the general method

2020-07-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149270#comment-17149270 ] jiaan.geng commented on SPARK-31846: According discussion between wuyi and me. It's not worth to do

[jira] [Commented] (SPARK-31844) DAGSchedulerSuite: For the pattern of failed + assert, extract the general method

2020-07-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149266#comment-17149266 ] jiaan.geng commented on SPARK-31844: According discussion between [~Ngone51] and me. It's not worth

[jira] [Commented] (SPARK-31842) DAGSchedulerSuite: For the pattern of runevent + assert, extract the general method

2020-07-01 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149265#comment-17149265 ] jiaan.geng commented on SPARK-31842: According discussion between [~Ngone51] and me. It's not worth

[jira] [Commented] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Bart Samwel (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149263#comment-17149263 ] Bart Samwel commented on SPARK-32130: - +1 to what [~cloud_fan] said. We should just keep the default

[jira] [Commented] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149246#comment-17149246 ] Wenchen Fan commented on SPARK-32130: - I'm not sure about 1. It's good to have but not necessary to

[jira] [Commented] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149235#comment-17149235 ] Maxim Gekk commented on SPARK-32130: I would like to propose: # Add the SQL config 

[jira] [Commented] (SPARK-30037) Customize krb5.conf to test Kafka delegation token with MiniKDC

2020-07-01 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149226#comment-17149226 ] angerszhu commented on SPARK-30037: --- These days I will work on this > Customize krb5.conf to test

[jira] [Commented] (SPARK-32130) Spark 3.0 json load performance is unacceptable in comparison of Spark 2.4

2020-07-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149225#comment-17149225 ] Wenchen Fan commented on SPARK-32130: - Even in Spark 2.4, the type inference takes much more time

[jira] [Resolved] (SPARK-32132) Thriftserver interval returns "4 weeks 2 days" in 2.4 and "30 days" in 3.0

2020-07-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32132. - Resolution: Not A Problem > Thriftserver interval returns "4 weeks 2 days" in 2.4 and "30 days"

[jira] [Updated] (SPARK-32146) ValueError when loading a PipelineModel on a personal computer

2020-07-01 Thread LoicH (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LoicH updated SPARK-32146: -- Description: I have a PipelineModel saved on my computer that I can't load using

[jira] [Comment Edited] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2020-07-01 Thread Jdub (Jira)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149212#comment-17149212 ] Jdub edited comment on SPARK-5594 at 7/1/20, 8:28 AM: -- If this can help, I had the

[jira] [Commented] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2020-07-01 Thread Jdub (Jira)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149212#comment-17149212 ] Jdub commented on SPARK-5594: - If this can help, I had the same error, but in a Firewalled environment where

[jira] [Updated] (SPARK-32146) ValueError when loading a PipelineModel on a personal computer

2020-07-01 Thread LoicH (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LoicH updated SPARK-32146: -- Description: I have a PipelineModel saved on my computer that I can't load using

[jira] [Updated] (SPARK-32146) ValueError when loading a PipelineModel on a personal computer

2020-07-01 Thread LoicH (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LoicH updated SPARK-32146: -- Description: I have a PipelineModel saved on my computer that I can't load using

[jira] [Updated] (SPARK-32147) Spark: PartitionBy changing the columns value

2020-07-01 Thread Shankar Koirala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shankar Koirala updated SPARK-32147: Description: While saving dataframe as parquet or csv with partitionBy column having 'f'

[jira] [Created] (SPARK-32147) Spark: PartitionBy changing the columns value

2020-07-01 Thread Shankar Koirala (Jira)
Shankar Koirala created SPARK-32147: --- Summary: Spark: PartitionBy changing the columns value Key: SPARK-32147 URL: https://issues.apache.org/jira/browse/SPARK-32147 Project: Spark Issue

[jira] [Updated] (SPARK-32146) ValueError when loading a PipelineModel on a personal computer

2020-07-01 Thread LoicH (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LoicH updated SPARK-32146: -- Description: I have a PipelineModel saved on my computer that I can't load using

[jira] [Updated] (SPARK-32146) ValueError when loading a PipelineModel on a personal computer

2020-07-01 Thread LoicH (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] LoicH updated SPARK-32146: -- Description: I have a PipelineModel saved on my computer that I can't load using

  1   2   >