[jira] [Created] (SPARK-46068) Improve error message when using a string literal where only an identifier can go

2023-11-22 Thread Serge Rielau (Jira)
Serge Rielau created SPARK-46068: Summary: Improve error message when using a string literal where only an identifier can go Key: SPARK-46068 URL: https://issues.apache.org/jira/browse/SPARK-46068

[jira] [Deleted] (SPARK-46049) Support groupingSets operation in PySpark (Spark Connect)

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon deleted SPARK-46049: - > Support groupingSets operation in PySpark (Spark Connect) >

[jira] [Updated] (SPARK-46067) Upgrade commons-compress to 1.25.0

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46067: --- Labels: pull-request-available (was: ) > Upgrade commons-compress to 1.25.0 >

[jira] [Resolved] (SPARK-46063) Improve error messages related to argument types in cute, rollup, groupby, and pivot

2023-11-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-46063. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43968

[jira] [Assigned] (SPARK-46063) Improve error messages related to argument types in cute, rollup, groupby, and pivot

2023-11-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-46063: - Assignee: Hyukjin Kwon > Improve error messages related to argument types in cute,

[jira] [Created] (SPARK-46067) Upgrade commons-compress to 1.25.0

2023-11-22 Thread Yang Jie (Jira)
Yang Jie created SPARK-46067: Summary: Upgrade commons-compress to 1.25.0 Key: SPARK-46067 URL: https://issues.apache.org/jira/browse/SPARK-46067 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-46060) Upgrade jdbc related test dependencies

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46060: --- Labels: pull-request-available (was: ) > Upgrade jdbc related test dependencies >

[jira] [Updated] (SPARK-46066) Use the Separators API instead of the String API to construct the DefaultPrettyPrinter

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46066: --- Labels: pull-request-available (was: ) > Use the Separators API instead of the String API

[jira] [Created] (SPARK-46066) Use the Separators API instead of the String API to construct the DefaultPrettyPrinter

2023-11-22 Thread Yang Jie (Jira)
Yang Jie created SPARK-46066: Summary: Use the Separators API instead of the String API to construct the DefaultPrettyPrinter Key: SPARK-46066 URL: https://issues.apache.org/jira/browse/SPARK-46066

[jira] [Updated] (SPARK-45862) Add user guide for basic dataframe operations

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45862: --- Labels: pull-request-available (was: ) > Add user guide for basic dataframe operations >

[jira] [Updated] (SPARK-46065) Refactor `(DataFrame|Series).factorize()` to use `create_map`.

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46065: --- Labels: pull-request-available (was: ) > Refactor `(DataFrame|Series).factorize()` to use

[jira] [Comment Edited] (SPARK-45311) Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"

2023-11-22 Thread Marc Le Bihan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788906#comment-17788906 ] Marc Le Bihan edited comment on SPARK-45311 at 11/23/23 6:08 AM: - I'll

[jira] [Comment Edited] (SPARK-45311) Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"

2023-11-22 Thread Marc Le Bihan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788906#comment-17788906 ] Marc Le Bihan edited comment on SPARK-45311 at 11/23/23 6:07 AM: - I'll

[jira] [Comment Edited] (SPARK-45311) Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"

2023-11-22 Thread Marc Le Bihan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788906#comment-17788906 ] Marc Le Bihan edited comment on SPARK-45311 at 11/23/23 6:07 AM: - I'll

[jira] [Created] (SPARK-46065) Refactor `(DataFrame|Series).factorize()` to use `create_map`.

2023-11-22 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46065: --- Summary: Refactor `(DataFrame|Series).factorize()` to use `create_map`. Key: SPARK-46065 URL: https://issues.apache.org/jira/browse/SPARK-46065 Project: Spark

[jira] [Updated] (SPARK-46063) Improve error messages related to argument types in cute, rollup, groupby, and pivot

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46063: --- Labels: pull-request-available (was: ) > Improve error messages related to argument types

[jira] [Updated] (SPARK-46063) Improve error messages related to argument types in cute, rollup, groupby, and pivot

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46063: - Description: {code} >>> spark.range(1).cube(cols=1.2) Traceback (most recent call last): File

[jira] [Updated] (SPARK-46063) Improve error messages related to argument types in cute, rollup, groupby, and pivot

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46063: - Summary: Improve error messages related to argument types in cute, rollup, groupby, and pivot

[jira] [Updated] (SPARK-46062) CTE reference node does not inherit the flag `isStreaming` from CTE definition node

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46062: --- Labels: pull-request-available (was: ) > CTE reference node does not inherit the flag

[jira] [Created] (SPARK-46063) Improve error messages related to argument types in cute, rollup, and pivot

2023-11-22 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-46063: Summary: Improve error messages related to argument types in cute, rollup, and pivot Key: SPARK-46063 URL: https://issues.apache.org/jira/browse/SPARK-46063 Project:

[jira] [Assigned] (SPARK-46061) Add the test party for reattach test case

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46061: Assignee: Hyukjin Kwon > Add the test party for reattach test case >

[jira] [Resolved] (SPARK-46061) Add the test party for reattach test case

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46061. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43965

[jira] [Resolved] (SPARK-46059) Install `six==1.16.0` explicitly for `pandas` in Python 3.12

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46059. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43964

[jira] [Commented] (SPARK-46062) CTE reference node does not inherit the flag `isStreaming` from CTE definition node

2023-11-22 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788953#comment-17788953 ] Jungtaek Lim commented on SPARK-46062: -- Working on the PR. Will submit a PR sooner. > CTE

[jira] [Created] (SPARK-46062) CTE reference node does not inherit the flag `isStreaming` from CTE definition node

2023-11-22 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-46062: Summary: CTE reference node does not inherit the flag `isStreaming` from CTE definition node Key: SPARK-46062 URL: https://issues.apache.org/jira/browse/SPARK-46062

[jira] [Resolved] (SPARK-45600) Make Python data source registration session level

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45600. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43742

[jira] [Assigned] (SPARK-45600) Make Python data source registration session level

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45600: Assignee: Allison Wang > Make Python data source registration session level >

[jira] [Commented] (SPARK-31347) Unable to run Spark Job on Federated Yarn Cluster, AMRMToken invalid

2023-11-22 Thread yanbin.zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788941#comment-17788941 ] yanbin.zhang commented on SPARK-31347: -- After enabling YARN Federation, I executed the command 

[jira] [Updated] (SPARK-46061) Add the test party for reattach test case

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46061: --- Labels: pull-request-available (was: ) > Add the test party for reattach test case >

[jira] [Created] (SPARK-46061) Add the test party for reattach test case

2023-11-22 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-46061: Summary: Add the test party for reattach test case Key: SPARK-46061 URL: https://issues.apache.org/jira/browse/SPARK-46061 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-46061) Add the test party for reattach test case

2023-11-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46061: - Issue Type: Test (was: New Feature) > Add the test party for reattach test case >

[jira] [Updated] (SPARK-46059) Install `six==1.16.0` explicitly for `pandas` in Python 3.12

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46059: -- Summary: Install `six==1.16.0` explicitly for `pandas` in Python 3.12 (was: Investigate

[jira] [Updated] (SPARK-46059) Investigate `pandas` import issues in Python 3.12 CI

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46059: --- Labels: pull-request-available (was: ) > Investigate `pandas` import issues in Python 3.12

[jira] [Created] (SPARK-46060) Upgrade jdbc related test dependencies

2023-11-22 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-46060: --- Summary: Upgrade jdbc related test dependencies Key: SPARK-46060 URL: https://issues.apache.org/jira/browse/SPARK-46060 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-46059) Investigate `pandas` import issues in Python 3.12 CI

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788934#comment-17788934 ] Dongjoon Hyun commented on SPARK-46059: --- `six` package is installed but `six.moves` is not visible

[jira] [Commented] (SPARK-46059) Investigate `pandas` import issues in Python 3.12 CI

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788933#comment-17788933 ] Dongjoon Hyun commented on SPARK-46059: --- It seems that I found the root case of the Infra docker

[jira] [Updated] (SPARK-44654) In subquery cannot perform partition pruning

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44654: --- Labels: performance pull-request-available (was: performance) > In subquery cannot perform

[jira] [Updated] (SPARK-44777) Allow to specify eagerness to RDD.checkpoint

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44777: --- Labels: pull-request-available (was: ) > Allow to specify eagerness to RDD.checkpoint >

[jira] [Commented] (SPARK-46059) Investigate `pandas` import issues in Python 3.12 CI

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788930#comment-17788930 ] Dongjoon Hyun commented on SPARK-46059: --- Although I use the same dependencies, this was not

[jira] [Created] (SPARK-46059) Investigate `pandas` import issues in Python 3.12 CI

2023-11-22 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46059: - Summary: Investigate `pandas` import issues in Python 3.12 CI Key: SPARK-46059 URL: https://issues.apache.org/jira/browse/SPARK-46059 Project: Spark Issue

[jira] [Assigned] (SPARK-46059) Investigate `pandas` import issues in Python 3.12 CI

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46059: - Assignee: Dongjoon Hyun > Investigate `pandas` import issues in Python 3.12 CI >

[jira] [Created] (SPARK-46058) [CORE] Add separate flag for privateKeyPassword

2023-11-22 Thread Hasnain Lakhani (Jira)
Hasnain Lakhani created SPARK-46058: --- Summary: [CORE] Add separate flag for privateKeyPassword Key: SPARK-46058 URL: https://issues.apache.org/jira/browse/SPARK-46058 Project: Spark Issue

[jira] [Comment Edited] (SPARK-45311) Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"

2023-11-22 Thread Marc Le Bihan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788906#comment-17788906 ] Marc Le Bihan edited comment on SPARK-45311 at 11/22/23 9:25 PM: - I'll

[jira] [Commented] (SPARK-45311) Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"

2023-11-22 Thread Marc Le Bihan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788906#comment-17788906 ] Marc Le Bihan commented on SPARK-45311: --- I'll do a try for 3.5.0. I believe that I was forced to

[jira] [Commented] (SPARK-45939) SPIP: Structured Streaming - Arbitrary State API v2

2023-11-22 Thread Anish Shrigondekar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788894#comment-17788894 ] Anish Shrigondekar commented on SPARK-45939: Added the docs here - SPIP:

[jira] [Commented] (SPARK-45311) Encoder fails on many "NoSuchElementException: None.get" since 3.4.x, search for an encoder for a generic type, and since 3.5.x isn't "an expression encoder"

2023-11-22 Thread Giambattista Bloisi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1779#comment-1779 ] Giambattista Bloisi commented on SPARK-45311: - I agree point 3) is a different problem that

[jira] [Updated] (SPARK-46056) Vectorized parquet reader throws NPE when reading files with DecimalType default values

2023-11-22 Thread Cosmin Dumitru (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Dumitru updated SPARK-46056: --- Affects Version/s: 4.0.0 > Vectorized parquet reader throws NPE when reading files with

[jira] [Updated] (SPARK-46057) Support SQL user-defined functions

2023-11-22 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-46057: - Description: This is an umbrella ticket to support SQL user-defined functions. (was: This is

[jira] [Created] (SPARK-46057) Support SQL user-defined functions

2023-11-22 Thread Allison Wang (Jira)
Allison Wang created SPARK-46057: Summary: Support SQL user-defined functions Key: SPARK-46057 URL: https://issues.apache.org/jira/browse/SPARK-46057 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-46056) Vectorized parquet reader throws NPE when reading files with DecimalType default values

2023-11-22 Thread Cosmin Dumitru (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Dumitru updated SPARK-46056: --- Description: The scenario is a bit more complicated than what the title says but it's not

[jira] [Updated] (SPARK-46056) Vectorized parquet reader throws NPE when reading files with DecimalType default values

2023-11-22 Thread Cosmin Dumitru (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Dumitru updated SPARK-46056: --- Description: The scenario is a bit more complicated than what the title says but it's not

[jira] [Updated] (SPARK-46056) Vectorized parquet reader throws NPE when reading files with DecimalType default values

2023-11-22 Thread Cosmin Dumitru (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Dumitru updated SPARK-46056: --- Description: The scenario is a bit more complicated than what the title says but it's not

[jira] [Updated] (SPARK-46056) Vectorized parquet reader throws NPE when reading files with DecimalType default values

2023-11-22 Thread Cosmin Dumitru (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Dumitru updated SPARK-46056: --- Description: The scenario is a bit more complicated than what the title says but it's not

[jira] [Created] (SPARK-46056) Vectorized parquet reader throws NPE when reading files with DecimalType default values

2023-11-22 Thread Cosmin Dumitru (Jira)
Cosmin Dumitru created SPARK-46056: -- Summary: Vectorized parquet reader throws NPE when reading files with DecimalType default values Key: SPARK-46056 URL: https://issues.apache.org/jira/browse/SPARK-46056

[jira] [Updated] (SPARK-46055) Refactor Catalog Database APIs implementation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46055: --- Labels: pull-request-available (was: ) > Refactor Catalog Database APIs implementation >

[jira] [Created] (SPARK-46055) Refactor Catalog Database APIs implementation

2023-11-22 Thread Yihong He (Jira)
Yihong He created SPARK-46055: - Summary: Refactor Catalog Database APIs implementation Key: SPARK-46055 URL: https://issues.apache.org/jira/browse/SPARK-46055 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46054) SPIP: Proposal to Adopt Google's Spark K8s Operator as Official Spark Operator

2023-11-22 Thread Vara Bonthu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vara Bonthu updated SPARK-46054: Summary: SPIP: Proposal to Adopt Google's Spark K8s Operator as Official Spark Operator (was:

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-22 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788743#comment-17788743 ] Bobby Wang commented on SPARK-46032: Hi [~gurwls223], BTW, I tried it on the fourth machine this

[jira] [Commented] (SPARK-46032) connect: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f

2023-11-22 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788717#comment-17788717 ] Bobby Wang commented on SPARK-46032: Hi [~gurwls223], Any suggestions? > connect: cannot assign

[jira] [Resolved] (SPARK-46042) Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46042. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43955

[jira] [Assigned] (SPARK-46042) Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46042: - Assignee: Dongjoon Hyun > Reenable a `releaseSession` test case in

[jira] [Assigned] (SPARK-46042) Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46042: - Assignee: Hyukjin Kwon (was: Dongjoon Hyun) > Reenable a `releaseSession` test case

[jira] [Resolved] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46053. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43957

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46053: - Assignee: Dongjoon Hyun > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Updated] (SPARK-46047) Fix the usage of Scala 2.13 deprecated APIs and make the usage of Scala 2.13 deprecated APIs as compilation error

2023-11-22 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-46047: - Summary: Fix the usage of Scala 2.13 deprecated APIs and make the usage of Scala 2.13 deprecated APIs

[jira] [Updated] (SPARK-46030) Spark set configuration parameters take a long time

2023-11-22 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46030: - Description: set spark.sql.sources.partitionOverwriteMode = dynamic; SLF4J: Failed to load class

[jira] [Resolved] (SPARK-46047) Remove the compilation suppression rule related to the usage of deprecated APIs in Scala 2.13

2023-11-22 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-46047. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43950

[jira] [Assigned] (SPARK-46047) Remove the compilation suppression rule related to the usage of deprecated APIs in Scala 2.13

2023-11-22 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-46047: Assignee: Yang Jie > Remove the compilation suppression rule related to the usage of deprecated

[jira] [Updated] (SPARK-46030) Spark set configuration parameters take a long time

2023-11-22 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46030: - Description: colored textset spark.sql.sources.partitionOverwriteMode = dynamic; SLF4J: Failed to load

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: (was: Apache Spark) > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: Apache Spark > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: Apache Spark > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: (was: Apache Spark) > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: Apache Spark > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: (was: Apache Spark) > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Assigned] (SPARK-46051) Cache python deps for linter and documentation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46051: -- Assignee: (was: Apache Spark) > Cache python deps for linter and documentation >

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: (was: Apache Spark) > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Assigned] (SPARK-46051) Cache python deps for linter and documentation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46051: -- Assignee: Apache Spark > Cache python deps for linter and documentation >

[jira] [Assigned] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46053: -- Assignee: Apache Spark > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Updated] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46053: --- Labels: pull-request-available (was: ) > Support DEDICATED_JVM_SBT_TESTS >

[jira] [Created] (SPARK-46053) Support DEDICATED_JVM_SBT_TESTS

2023-11-22 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46053: - Summary: Support DEDICATED_JVM_SBT_TESTS Key: SPARK-46053 URL: https://issues.apache.org/jira/browse/SPARK-46053 Project: Spark Issue Type: Test

[jira] [Assigned] (SPARK-46051) Cache python deps for linter and documentation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46051: -- Assignee: (was: Apache Spark) > Cache python deps for linter and documentation >

[jira] [Assigned] (SPARK-46051) Cache python deps for linter and documentation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46051: -- Assignee: Apache Spark > Cache python deps for linter and documentation >

[jira] [Assigned] (SPARK-46051) Cache python deps for linter and documentation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46051: -- Assignee: Apache Spark > Cache python deps for linter and documentation >

[jira] [Assigned] (SPARK-46051) Cache python deps for linter and documentation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46051: -- Assignee: (was: Apache Spark) > Cache python deps for linter and documentation >

[jira] [Updated] (SPARK-46042) Reenable a `releaseSession` test case in SparkConnectServiceE2ESuite

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46042: --- Labels: pull-request-available (was: ) > Reenable a `releaseSession` test case in

[jira] [Resolved] (SPARK-46006) YarnAllocator miss clean targetNumExecutorsPerResourceProfileId after YarnSchedulerBackend call stop

2023-11-22 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46006. -- Fix Version/s: 3.3.4 Resolution: Fixed Issue resolved by pull request 43906

[jira] [Assigned] (SPARK-46006) YarnAllocator miss clean targetNumExecutorsPerResourceProfileId after YarnSchedulerBackend call stop

2023-11-22 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-46006: Assignee: angerszhu > YarnAllocator miss clean targetNumExecutorsPerResourceProfileId after >

[jira] [Updated] (SPARK-46052) Remove unnecessary TaskScheduler.killAllTaskAttempts

2023-11-22 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-46052: - Description: Spark has two functions to kill all tasks in a Stage: * `cancelTasks`: Not only kill all the

[jira] [Updated] (SPARK-46052) Remove unnecessary TaskScheduler.killAllTaskAttempts

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46052: --- Labels: pull-request-available (was: ) > Remove unnecessary

[jira] [Created] (SPARK-46052) Remove unnecessary TaskScheduler.killAllTaskAttempts

2023-11-22 Thread wuyi (Jira)
wuyi created SPARK-46052: Summary: Remove unnecessary TaskScheduler.killAllTaskAttempts Key: SPARK-46052 URL: https://issues.apache.org/jira/browse/SPARK-46052 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-46051) Cache python deps for linter and documentation

2023-11-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46051: --- Labels: pull-request-available (was: ) > Cache python deps for linter and documentation >