[jira] [Resolved] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx resolved SPARK-42912. -- Resolution: Fixed > Some cases do not take effect when using OptimizeSkewInRebalancePartitions > -

[jira] [Closed] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx closed SPARK-42912. > Some cases do not take effect when using OptimizeSkewInRebalancePartitions > ---

[jira] [Comment Edited] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704479#comment-17704479 ] thomasgx edited comment on SPARK-42912 at 3/24/23 6:52 AM: --- fo

[jira] [Commented] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704479#comment-17704479 ] thomasgx commented on SPARK-42912: -- https://issues.apache.org/jira/browse/SPARK-36967

[jira] [Resolved] (SPARK-42861) Review and fix issues in SQL API docs

2023-03-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-42861. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40541 [https://github.com

[jira] [Resolved] (SPARK-42904) Char/Varchar Support for JDBC Catalog

2023-03-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-42904. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40531 [https://github.com

[jira] [Assigned] (SPARK-42904) Char/Varchar Support for JDBC Catalog

2023-03-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-42904: Assignee: Kent Yao > Char/Varchar Support for JDBC Catalog >

[jira] [Commented] (SPARK-42861) Review and fix issues in SQL API docs

2023-03-23 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704462#comment-17704462 ] Snoot.io commented on SPARK-42861: -- User 'cloud-fan' has created a pull request for thi

[jira] [Commented] (SPARK-42913) Upgrade Hadoop to 3.3.5

2023-03-23 Thread Snoot.io (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704458#comment-17704458 ] Snoot.io commented on SPARK-42913: -- User 'LuciferYang' has created a pull request for t

[jira] [Created] (SPARK-42914) Reuse transformUnregisteredFunction for DistributedSequenceID.

2023-03-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-42914: --- Summary: Reuse transformUnregisteredFunction for DistributedSequenceID. Key: SPARK-42914 URL: https://issues.apache.org/jira/browse/SPARK-42914 Project: Spark

[jira] [Created] (SPARK-42913) Upgrade Hadoop to 3.3.5

2023-03-23 Thread Yang Jie (Jira)
Yang Jie created SPARK-42913: Summary: Upgrade Hadoop to 3.3.5 Key: SPARK-42913 URL: https://issues.apache.org/jira/browse/SPARK-42913 Project: Spark Issue Type: Improvement Components:

[jira] [Commented] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704440#comment-17704440 ] thomasgx commented on SPARK-42912: -- Can anyone help me explain this problem > Some cas

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Attachment: image-2023-03-24-11-37-42-289.png > Some cases do not take effect when using OptimizeSkewInR

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Description: Questioin: When using OptimizeSkewInRebalancePartitions to insert dynamic partitions (thr

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Attachment: image-2023-03-24-11-36-54-539.png > Some cases do not take effect when using OptimizeSkewInR

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Description: Questioin: When using OptimizeSkewInRebalancePartitions to insert dynamic partitions (thr

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Attachment: image-2023-03-24-11-34-34-070.png > Some cases do not take effect when using OptimizeSkewInR

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Description: Questioin: When using OptimizeSkewInRebalancePartitions to insert dynamic partitions (thr

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Description: Questioin: When using OptimizeSkewInRebalancePartitions to insert dynamic partitions (thr

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Attachment: image-2023-03-24-11-31-42-564.png > Some cases do not take effect when using OptimizeSkewInR

[jira] [Updated] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] thomasgx updated SPARK-42912: - Attachment: image-2023-03-24-11-30-42-239.png > Some cases do not take effect when using OptimizeSkewInR

[jira] [Created] (SPARK-42912) Some cases do not take effect when using OptimizeSkewInRebalancePartitions

2023-03-23 Thread thomasgx (Jira)
thomasgx created SPARK-42912: Summary: Some cases do not take effect when using OptimizeSkewInRebalancePartitions Key: SPARK-42912 URL: https://issues.apache.org/jira/browse/SPARK-42912 Project: Spark

[jira] [Updated] (SPARK-42693) API Auditing

2023-03-23 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42693: - Description: Audit user-facing API of Spark 3.4. The main goal is to ensure public API docs to

[jira] [Updated] (SPARK-42693) API Auditing

2023-03-23 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42693: - Description: Audit user-facing API of Spark 3.4. The main goal is to ensure public API docs to

[jira] [Updated] (SPARK-42905) pyspark.ml.stat.Correlation - Spearman Correlation method giving incorrect and inconsistent results for the same DataFrame if it has huge amount of Ties.

2023-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-42905: - Priority: Critical (was: Blocker) > pyspark.ml.stat.Correlation - Spearman Correlation method g

[jira] [Commented] (SPARK-42910) Generic annotation of class attribute in abstract class is NOT initalized in inherited classes

2023-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704403#comment-17704403 ] Hyukjin Kwon commented on SPARK-42910: -- cc [~zero323] FYI > Generic annotation of

[jira] [Resolved] (SPARK-42909) INSERT INTO with column list does not work

2023-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42909. -- Resolution: Duplicate > INSERT INTO with column list does not work > -

[jira] [Created] (SPARK-42911) Introduce more basic exceptions.

2023-03-23 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42911: - Summary: Introduce more basic exceptions. Key: SPARK-42911 URL: https://issues.apache.org/jira/browse/SPARK-42911 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-42910) Generic annotation of class attribute in abstract class is NOT initalized in inherited classes

2023-03-23 Thread Jon Farzanfar (Jira)
Jon Farzanfar created SPARK-42910: - Summary: Generic annotation of class attribute in abstract class is NOT initalized in inherited classes Key: SPARK-42910 URL: https://issues.apache.org/jira/browse/SPARK-42910

[jira] [Commented] (SPARK-42909) INSERT INTO with column list does not work

2023-03-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704368#comment-17704368 ] Bruce Robbins commented on SPARK-42909: --- It looks like this capability landed in 3

[jira] [Commented] (SPARK-39821) DatetimeIndex error during pyspark session

2023-03-23 Thread Ihor Bobak (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704234#comment-17704234 ] Ihor Bobak commented on SPARK-39821: I have Pandas version 1.5.3 and numpy version

[jira] [Reopened] (SPARK-41537) Protobuf backwards compatibility testing

2023-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-41537: -- Assignee: (was: Rui Wang) Reverted in https://github.com/apache/spark/commit/04202b3fdd

[jira] [Updated] (SPARK-41537) Protobuf backwards compatibility testing

2023-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41537: - Fix Version/s: (was: 3.5.0) > Protobuf backwards compatibility testing > ---

[jira] [Resolved] (SPARK-41537) Protobuf backwards compatibility testing

2023-03-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41537. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 39294 [https://gi

[jira] [Created] (SPARK-42909) INSERT INTO with column list does not work

2023-03-23 Thread Tjomme Vergauwen (Jira)
Tjomme Vergauwen created SPARK-42909: Summary: INSERT INTO with column list does not work Key: SPARK-42909 URL: https://issues.apache.org/jira/browse/SPARK-42909 Project: Spark Issue Type

[jira] [Commented] (SPARK-42584) Improve output of Column.explain

2023-03-23 Thread Nikita Awasthi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704091#comment-17704091 ] Nikita Awasthi commented on SPARK-42584: User 'beliefer' has created a pull requ

[jira] [Updated] (SPARK-40513) SPIP: Support Docker Official Image for Spark

2023-03-23 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-40513: Affects Version/s: 3.5.0 (was: 3.4.0) > SPIP: Support Docker Official I

[jira] [Commented] (SPARK-40513) SPIP: Support Docker Official Image for Spark

2023-03-23 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704085#comment-17704085 ] Yikun Jiang commented on SPARK-40513: - Due to https://github.com/docker-library/offi

[jira] [Commented] (SPARK-42907) Implement Avro functions

2023-03-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703997#comment-17703997 ] ASF GitHub Bot commented on SPARK-42907: User 'zhengruifeng' has created a pull

[jira] [Commented] (SPARK-42906) Resource name prefix should start with an alphabetic character

2023-03-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703996#comment-17703996 ] ASF GitHub Bot commented on SPARK-42906: User 'pan3793' has created a pull reque

[jira] [Updated] (SPARK-42551) Support more subexpression elimination cases

2023-03-23 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-42551: Description: h1. *Design Sketch* * Get all common expressions from input expressions. Recursively visits

[jira] [Updated] (SPARK-42907) Implement Avro functions

2023-03-23 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-42907: -- Parent: SPARK-41283 Issue Type: Sub-task (was: New Feature) > Implement Avro function

[jira] [Assigned] (SPARK-42900) Fix createDataFrame to respect both type inference and column names.

2023-03-23 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-42900: - Assignee: Takuya Ueshin > Fix createDataFrame to respect both type inference and column

[jira] [Resolved] (SPARK-42900) Fix createDataFrame to respect both type inference and column names.

2023-03-23 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-42900. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40527 [https://

[jira] [Created] (SPARK-42908) Raise RuntimeError if SparkContext is not initialized when parsing DDL-formatted type strings

2023-03-23 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-42908: Summary: Raise RuntimeError if SparkContext is not initialized when parsing DDL-formatted type strings Key: SPARK-42908 URL: https://issues.apache.org/jira/browse/SPARK-42908

[jira] [Created] (SPARK-42907) Implement Avro functions

2023-03-23 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-42907: - Summary: Implement Avro functions Key: SPARK-42907 URL: https://issues.apache.org/jira/browse/SPARK-42907 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-42905) pyspark.ml.stat.Correlation - Spearman Correlation method giving incorrect and inconsistent results for the same DataFrame if it has huge amount of Ties.

2023-03-23 Thread dronzer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dronzer updated SPARK-42905: Description: pyspark.ml.stat.Correlation Following is the Scenario where the Correlation function fails f

[jira] [Commented] (SPARK-42903) Avoid documenting None as as a return value in docstring

2023-03-23 Thread Mike K (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703950#comment-17703950 ] Mike K commented on SPARK-42903: User 'xinrong-meng' has created a pull request for this

[jira] [Created] (SPARK-42906) Resource name prefix should start with an alphabetic character

2023-03-23 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-42906: - Summary: Resource name prefix should start with an alphabetic character Key: SPARK-42906 URL: https://issues.apache.org/jira/browse/SPARK-42906 Project: Spark Iss