[jira] [Resolved] (SPARK-38623) Add more comments and tests for HashShuffleSpec

2022-03-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38623. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35937

[jira] [Assigned] (SPARK-38623) Add more comments and tests for HashShuffleSpec

2022-03-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38623: --- Assignee: Wenchen Fan > Add more comments and tests for HashShuffleSpec >

[jira] [Assigned] (SPARK-38671) Publish snapshot from branch-3.3

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38671: Assignee: Apache Spark > Publish snapshot from branch-3.3 >

[jira] [Commented] (SPARK-38671) Publish snapshot from branch-3.3

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513151#comment-17513151 ] Apache Spark commented on SPARK-38671: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38671) Publish snapshot from branch-3.3

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38671: Assignee: (was: Apache Spark) > Publish snapshot from branch-3.3 >

[jira] [Created] (SPARK-38672) Kafka, spark data frame read fails when a message does not exist for the timestamp in all partitions

2022-03-27 Thread gattu shivakumar (Jira)
gattu shivakumar created SPARK-38672: Summary: Kafka, spark data frame read fails when a message does not exist for the timestamp in all partitions Key: SPARK-38672 URL:

[jira] [Resolved] (SPARK-38391) Datasource v2 supports partial topN push-down

2022-03-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38391. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35710

[jira] [Assigned] (SPARK-38391) Datasource v2 supports partial topN push-down

2022-03-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38391: --- Assignee: jiaan.geng > Datasource v2 supports partial topN push-down >

[jira] [Commented] (SPARK-38671) Publish snapshot from branch-3.3

2022-03-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513142#comment-17513142 ] Yuming Wang commented on SPARK-38671: - https://github.com/apache/spark/pull/35986 > Publish

[jira] [Created] (SPARK-38671) Publish snapshot from branch-3.3

2022-03-27 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38671: --- Summary: Publish snapshot from branch-3.3 Key: SPARK-38671 URL: https://issues.apache.org/jira/browse/SPARK-38671 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-38616) Keep track of SQL query text in Catalyst TreeNode

2022-03-27 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-38616. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35936

[jira] [Assigned] (SPARK-38670) Add offset commit time to streaming query listener

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38670: Assignee: (was: Apache Spark) > Add offset commit time to streaming query listener >

[jira] [Assigned] (SPARK-38670) Add offset commit time to streaming query listener

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38670: Assignee: Apache Spark > Add offset commit time to streaming query listener >

[jira] [Commented] (SPARK-38670) Add offset commit time to streaming query listener

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513112#comment-17513112 ] Apache Spark commented on SPARK-38670: -- User 'jerrypeng' has created a pull request for this issue:

[jira] [Created] (SPARK-38670) Add offset commit time to streaming query listener

2022-03-27 Thread Boyang Jerry Peng (Jira)
Boyang Jerry Peng created SPARK-38670: - Summary: Add offset commit time to streaming query listener Key: SPARK-38670 URL: https://issues.apache.org/jira/browse/SPARK-38670 Project: Spark

[jira] [Assigned] (SPARK-38669) Validate input dataset of ml.clustering

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38669: Assignee: Apache Spark > Validate input dataset of ml.clustering >

[jira] [Commented] (SPARK-38669) Validate input dataset of ml.clustering

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513099#comment-17513099 ] Apache Spark commented on SPARK-38669: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-38669) Validate input dataset of ml.clustering

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38669: Assignee: (was: Apache Spark) > Validate input dataset of ml.clustering >

[jira] [Created] (SPARK-38669) Validate input dataset of ml.clustering

2022-03-27 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-38669: Summary: Validate input dataset of ml.clustering Key: SPARK-38669 URL: https://issues.apache.org/jira/browse/SPARK-38669 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-38336) Catalyst changes for DEFAULT column support

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513061#comment-17513061 ] Apache Spark commented on SPARK-38336: -- User 'dtenedor' has created a pull request for this issue:

[jira] [Commented] (SPARK-38336) Catalyst changes for DEFAULT column support

2022-03-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513060#comment-17513060 ] Apache Spark commented on SPARK-38336: -- User 'dtenedor' has created a pull request for this issue:

[jira] [Updated] (SPARK-38668) Spark on Kubernetes: add separate pod watcher service to reduce pressure on K8s API server

2022-03-27 Thread BoYang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BoYang updated SPARK-38668: --- Summary: Spark on Kubernetes: add separate pod watcher service to reduce pressure on K8s API server (was:

[jira] [Updated] (SPARK-38668) Spark on Kubernetes: support external pod watcher to reduce pressure on K8s API server

2022-03-27 Thread BoYang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BoYang updated SPARK-38668: --- Description: Spark driver will listen to all pods events to manage its executor pods. This will cause

[jira] [Created] (SPARK-38668) Spark on Kubernetes: support external pod watcher to reduce pressure on K8s API server

2022-03-27 Thread BoYang (Jira)
BoYang created SPARK-38668: -- Summary: Spark on Kubernetes: support external pod watcher to reduce pressure on K8s API server Key: SPARK-38668 URL: https://issues.apache.org/jira/browse/SPARK-38668 Project:

[jira] [Updated] (SPARK-38667) Optimizer generates error when using inner join along with sequence

2022-03-27 Thread Lars (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars updated SPARK-38667: - Description: This issue occurred in a more complex scenario, so I've broken it down into a simple case.

[jira] [Created] (SPARK-38667) Optimizer generates error when using inner join along with sequence

2022-03-27 Thread Lars (Jira)
Lars created SPARK-38667: Summary: Optimizer generates error when using inner join along with sequence Key: SPARK-38667 URL: https://issues.apache.org/jira/browse/SPARK-38667 Project: Spark Issue

[jira] [Resolved] (SPARK-38665) upgrade jackson due to CVE-2020-36518

2022-03-27 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-38665. - Fix Version/s: 3.3.0 Assignee: Cheng Pan Resolution: Fixed Issue resolved by

[jira] [Commented] (SPARK-19335) Spark should support doing an efficient DataFrame Upsert via JDBC

2022-03-27 Thread s4782 (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512998#comment-17512998 ] s4782 commented on SPARK-19335: --- Hi,  is there any update on when this will be added?   > Spark should

[jira] [Updated] (SPARK-38664) Support compact EventLog when there are illegal characters in the path

2022-03-27 Thread w l (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] w l updated SPARK-38664: Summary: Support compact EventLog when there are illegal characters in the path (was: EventLog compact will fail

[jira] [Updated] (SPARK-38257) Upgrade rocksdbjni to 7.0.3

2022-03-27 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-38257: - Summary: Upgrade rocksdbjni to 7.0.3 (was: Upgrade rocksdbjni to 6.29.3) > Upgrade rocksdbjni to 7.0.3

[jira] [Commented] (SPARK-38652) K8S IT Test DepsTestsSuite blocks with PathIOException in hadoop-aws-3.3.2

2022-03-27 Thread qian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512846#comment-17512846 ] qian commented on SPARK-38652: -- [~ste...@apache.org] Hi, I run the same suite against an real aws s3

[jira] [Updated] (SPARK-38664) EventLog compact will fail if there are illegal characters in the path

2022-03-27 Thread w l (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] w l updated SPARK-38664: Description: When a path has illegal characters, it is escaped.  The file will not be found and compactor  will