[jira] [Assigned] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38077: Assignee: (was: Apache Spark) > Spark 3.2.1 breaks binary compatibility with Spark

[jira] [Assigned] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38077: Assignee: Apache Spark > Spark 3.2.1 breaks binary compatibility with Spark 3.2.0 >

[jira] [Commented] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485072#comment-17485072 ] Apache Spark commented on SPARK-38077: -- User 'sunchao' has created a pull request for this issue:

[jira] [Commented] (SPARK-37958) Pyspark SparkContext.AddFile() does not respect spark.files.overwrite

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485047#comment-17485047 ] Apache Spark commented on SPARK-37958: -- User 'yoda-mon' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37958) Pyspark SparkContext.AddFile() does not respect spark.files.overwrite

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37958: Assignee: (was: Apache Spark) > Pyspark SparkContext.AddFile() does not respect

[jira] [Assigned] (SPARK-37958) Pyspark SparkContext.AddFile() does not respect spark.files.overwrite

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37958: Assignee: Apache Spark > Pyspark SparkContext.AddFile() does not respect

[jira] [Commented] (SPARK-37958) Pyspark SparkContext.AddFile() does not respect spark.files.overwrite

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485046#comment-17485046 ] Apache Spark commented on SPARK-37958: -- User 'yoda-mon' has created a pull request for this issue:

[jira] [Updated] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38081: -- Parent: (was: SPARK-33772) Issue Type: Test (was: Sub-task) > Support

[jira] [Commented] (SPARK-37995) TPCDS 1TB q72 fails when spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly is false

2022-01-31 Thread Kapil Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485039#comment-17485039 ] Kapil Singh commented on SPARK-37995: - Hey folks, any input on this issue? [~hyukjin.kwon]

[jira] [Commented] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485037#comment-17485037 ] Apache Spark commented on SPARK-38081: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485036#comment-17485036 ] Apache Spark commented on SPARK-38081: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38081: Assignee: (was: Apache Spark) > Support cloud-backend in K8s IT with SBT >

[jira] [Assigned] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38081: Assignee: Apache Spark > Support cloud-backend in K8s IT with SBT >

[jira] [Created] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-01-31 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38081: - Summary: Support cloud-backend in K8s IT with SBT Key: SPARK-38081 URL: https://issues.apache.org/jira/browse/SPARK-38081 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37958) Pyspark SparkContext.AddFile() does not respect spark.files.overwrite

2022-01-31 Thread Leona Yoda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485032#comment-17485032 ] Leona Yoda commented on SPARK-37958: This is probably default behavior not only on k8s or pyspark

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38075: -- Fix Version/s: 3.1.3 > Hive script transform with order by and limit will return fake rows >

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-01-31 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-01-31 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Updated] (SPARK-38063) Support SQL split_part function

2022-01-31 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-38063: - Description: `split_part()` is a commonly supported function by other systems such as Postgres and

[jira] [Commented] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484997#comment-17484997 ] Apache Spark commented on SPARK-38075: -- User 'bersprockets' has created a pull request for this

[jira] [Commented] (SPARK-34183) DataSource V2: Support required distribution and ordering in SS

2022-01-31 Thread Anton Okolnychyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484985#comment-17484985 ] Anton Okolnychyi commented on SPARK-34183: -- I am sorry it took me a while to come back to this.

[jira] [Commented] (SPARK-34183) DataSource V2: Support required distribution and ordering in SS

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484982#comment-17484982 ] Apache Spark commented on SPARK-34183: -- User 'aokolnychyi' has created a pull request for this

[jira] [Commented] (SPARK-34183) DataSource V2: Support required distribution and ordering in SS

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484981#comment-17484981 ] Apache Spark commented on SPARK-34183: -- User 'aokolnychyi' has created a pull request for this

[jira] [Comment Edited] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Nadav Samet (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484941#comment-17484941 ] Nadav Samet edited comment on SPARK-38077 at 1/31/22, 9:14 PM: --- I wasn't

[jira] [Commented] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Nadav Samet (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484941#comment-17484941 ] Nadav Samet commented on SPARK-38077: - I wasn't aware that the policy allows for it, but I'd like to

[jira] [Assigned] (SPARK-38047) Provide an option to only roll executors if they are outliers

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38047: Assignee: Apache Spark (was: Alex Holmes) > Provide an option to only roll executors if

[jira] [Assigned] (SPARK-38047) Provide an option to only roll executors if they are outliers

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38047: Assignee: Alex Holmes (was: Apache Spark) > Provide an option to only roll executors if

[jira] [Commented] (SPARK-38047) Provide an option to only roll executors if they are outliers

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484933#comment-17484933 ] Apache Spark commented on SPARK-38047: -- User 'alexholmes' has created a pull request for this

[jira] [Commented] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484894#comment-17484894 ] Chao Sun commented on SPARK-38077: -- BTW [~thesamet] it seems Spark only guarantees API compatibility,

[jira] [Commented] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484893#comment-17484893 ] Apache Spark commented on SPARK-38080: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38080: Assignee: (was: Apache Spark) > Flaky test: StreamingQueryManagerSuite:

[jira] [Assigned] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38080: Assignee: Apache Spark > Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination

[jira] [Commented] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484891#comment-17484891 ] Apache Spark commented on SPARK-38080: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Updated] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-38080: - Description: {code:java} [info] - awaitAnyTermination with timeout and resetTerminated ***

[jira] [Commented] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484873#comment-17484873 ] Chao Sun commented on SPARK-38077: -- Sorry for breaking the binary compatibility. I wasn't aware that

[jira] [Updated] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38077: -- Affects Version/s: (was: 3.2.0) > Spark 3.2.1 breaks binary compatibility with Spark

[jira] [Commented] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-01-31 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484865#comment-17484865 ] Erik Krogen commented on SPARK-38062: - Thanks for taking a look btw! Even if we don't agree on

[jira] [Commented] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-01-31 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484864#comment-17484864 ] Erik Krogen commented on SPARK-38062: - It still seems like a bug to me that the presence of a DNS

[jira] [Commented] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484863#comment-17484863 ] Dongjoon Hyun commented on SPARK-38077: --- cc [~sunchao] > Spark 3.2.1 breaks binary compatibility

[jira] [Updated] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-38080: - Description: {code:java} [info] - awaitAnyTermination with timeout and resetTerminated ***

[jira] [Resolved] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38075. --- Fix Version/s: 3.3.0 3.2.2 Assignee: Bruce Robbins

[jira] [Commented] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484855#comment-17484855 ] Dongjoon Hyun commented on SPARK-38062: --- I agree that you may think like that, but I don't think

[jira] [Updated] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38062: -- Affects Version/s: 3.3.0 (was: 3.1.1) > FallbackStorage shouldn't

[jira] [Updated] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-01-31 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38062: -- Issue Type: Improvement (was: Bug) > FallbackStorage shouldn't attempt to resolve arbitrary

[jira] [Created] (SPARK-38080) Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated'

2022-01-31 Thread Shixiong Zhu (Jira)
Shixiong Zhu created SPARK-38080: Summary: Flaky test: StreamingQueryManagerSuite: 'awaitAnyTermination with timeout and resetTerminated' Key: SPARK-38080 URL: https://issues.apache.org/jira/browse/SPARK-38080

[jira] [Updated] (SPARK-38079) Not waiting for configmap before starting driver

2022-01-31 Thread Ben (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben updated SPARK-38079: Description: *The problem* When you spark-submit to kubernetes in cluster-mode: # Kubernetes creates the driver

[jira] [Created] (SPARK-38079) Not waiting for configmap before starting driver

2022-01-31 Thread Ben (Jira)
Ben created SPARK-38079: --- Summary: Not waiting for configmap before starting driver Key: SPARK-38079 URL: https://issues.apache.org/jira/browse/SPARK-38079 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-01-31 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484806#comment-17484806 ] Erik Krogen edited comment on SPARK-38062 at 1/31/22, 5:33 PM: --- I think

[jira] [Commented] (SPARK-38062) FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname

2022-01-31 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484806#comment-17484806 ] Erik Krogen commented on SPARK-38062: - I think the issue I'm reporting is very similar to

[jira] [Created] (SPARK-38078) Aggregation with Watermark in AppendMode is holding data beyong water mark boundary.

2022-01-31 Thread krishna (Jira)
krishna created SPARK-38078: --- Summary: Aggregation with Watermark in AppendMode is holding data beyong water mark boundary. Key: SPARK-38078 URL: https://issues.apache.org/jira/browse/SPARK-38078 Project:

[jira] [Created] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Nadav Samet (Jira)
Nadav Samet created SPARK-38077: --- Summary: Spark 3.2.1 breaks binary compatibility with Spark 3.2.0 Key: SPARK-38077 URL: https://issues.apache.org/jira/browse/SPARK-38077 Project: Spark Issue

[jira] [Updated] (SPARK-38077) Spark 3.2.1 breaks binary compatibility with Spark 3.2.0

2022-01-31 Thread Nadav Samet (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nadav Samet updated SPARK-38077: Description: [PR 35243|https://github.com/apache/spark/pull/35243] introduced a new parameter to

[jira] [Assigned] (SPARK-37946) Use error classes in the execution errors related to partitions

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37946: Assignee: Apache Spark > Use error classes in the execution errors related to partitions

[jira] [Assigned] (SPARK-37946) Use error classes in the execution errors related to partitions

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37946: Assignee: (was: Apache Spark) > Use error classes in the execution errors related to

[jira] [Commented] (SPARK-37946) Use error classes in the execution errors related to partitions

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484592#comment-17484592 ] Apache Spark commented on SPARK-37946: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37946) Use error classes in the execution errors related to partitions

2022-01-31 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37946: Assignee: Apache Spark > Use error classes in the execution errors related to partitions