[jira] [Resolved] (SPARK-48335) Make `_parse_datatype_string` compatible with Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-48335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48335. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46657 [https://github.com/apache/spark/pull/46657] > Make `_parse_datatype_string` compatible with Spark Connect > --- > > Key: SPARK-48335 > URL: https://issues.apache.org/jira/browse/SPARK-48335 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48335) Make `_parse_datatype_string` compatible with Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-48335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-48335: - Assignee: Ruifeng Zheng > Make `_parse_datatype_string` compatible with Spark Connect > --- > > Key: SPARK-48335 > URL: https://issues.apache.org/jira/browse/SPARK-48335 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48335) Make `_parse_datatype_string` compatible with Spark Connect
Ruifeng Zheng created SPARK-48335: - Summary: Make `_parse_datatype_string` compatible with Spark Connect Key: SPARK-48335 URL: https://issues.apache.org/jira/browse/SPARK-48335 Project: Spark Issue Type: Bug Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48335) Make `_parse_datatype_string` compatible with Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-48335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-48335: -- Issue Type: Improvement (was: Bug) > Make `_parse_datatype_string` compatible with Spark Connect > --- > > Key: SPARK-48335 > URL: https://issues.apache.org/jira/browse/SPARK-48335 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48333) Test `test_sorting_functions_with_column` with same `Column`
Ruifeng Zheng created SPARK-48333: - Summary: Test `test_sorting_functions_with_column` with same `Column` Key: SPARK-48333 URL: https://issues.apache.org/jira/browse/SPARK-48333 Project: Spark Issue Type: Sub-task Components: Connect, PySpark, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48321) Avoid using deprecated methods in dsl
[ https://issues.apache.org/jira/browse/SPARK-48321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48321. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46635 [https://github.com/apache/spark/pull/46635] > Avoid using deprecated methods in dsl > - > > Key: SPARK-48321 > URL: https://issues.apache.org/jira/browse/SPARK-48321 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48322) Drop internal metadata in `DataFrame.schema`
Ruifeng Zheng created SPARK-48322: - Summary: Drop internal metadata in `DataFrame.schema` Key: SPARK-48322 URL: https://issues.apache.org/jira/browse/SPARK-48322 Project: Spark Issue Type: Improvement Components: Connect, PySpark, SQL Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48321) Avoid using deprecated methods in dsl
Ruifeng Zheng created SPARK-48321: - Summary: Avoid using deprecated methods in dsl Key: SPARK-48321 URL: https://issues.apache.org/jira/browse/SPARK-48321 Project: Spark Issue Type: Improvement Components: Connect Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48319) Test `assert_true` and `raise_error` with more specific error class
Ruifeng Zheng created SPARK-48319: - Summary: Test `assert_true` and `raise_error` with more specific error class Key: SPARK-48319 URL: https://issues.apache.org/jira/browse/SPARK-48319 Project: Spark Issue Type: Sub-task Components: Connect, PySpark, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48319) Test `assert_true` and `raise_error` with the same error class as Spark Classic
[ https://issues.apache.org/jira/browse/SPARK-48319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-48319: -- Summary: Test `assert_true` and `raise_error` with the same error class as Spark Classic (was: Test `assert_true` and `raise_error` with more specific error class) > Test `assert_true` and `raise_error` with the same error class as Spark > Classic > --- > > Key: SPARK-48319 > URL: https://issues.apache.org/jira/browse/SPARK-48319 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48301) Rename CREATE_FUNC_WITH_IF_NOT_EXISTS_AND_REPLACE to CREATE_ROUTINE_WITH_IF_NOT_EXISTS_AND_REPLACE
[ https://issues.apache.org/jira/browse/SPARK-48301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48301. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46608 [https://github.com/apache/spark/pull/46608] > Rename CREATE_FUNC_WITH_IF_NOT_EXISTS_AND_REPLACE to > CREATE_ROUTINE_WITH_IF_NOT_EXISTS_AND_REPLACE > -- > > Key: SPARK-48301 > URL: https://issues.apache.org/jira/browse/SPARK-48301 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48287) Apply the builtin `timestamp_diff` method
[ https://issues.apache.org/jira/browse/SPARK-48287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-48287: - Assignee: Ruifeng Zheng > Apply the builtin `timestamp_diff` method > - > > Key: SPARK-48287 > URL: https://issues.apache.org/jira/browse/SPARK-48287 > Project: Spark > Issue Type: Improvement > Components: Connect, PS >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48287) Apply the builtin `timestamp_diff` method
[ https://issues.apache.org/jira/browse/SPARK-48287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48287. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46595 [https://github.com/apache/spark/pull/46595] > Apply the builtin `timestamp_diff` method > - > > Key: SPARK-48287 > URL: https://issues.apache.org/jira/browse/SPARK-48287 > Project: Spark > Issue Type: Improvement > Components: Connect, PS >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48295) Turn on compute.ops_on_diff_frames by default
Ruifeng Zheng created SPARK-48295: - Summary: Turn on compute.ops_on_diff_frames by default Key: SPARK-48295 URL: https://issues.apache.org/jira/browse/SPARK-48295 Project: Spark Issue Type: Improvement Components: PS Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48287) Apply the builtin `timestamp_diff` method
Ruifeng Zheng created SPARK-48287: - Summary: Apply the builtin `timestamp_diff` method Key: SPARK-48287 URL: https://issues.apache.org/jira/browse/SPARK-48287 Project: Spark Issue Type: Improvement Components: Connect, PS Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48278) Refine the string representation of `Cast`
[ https://issues.apache.org/jira/browse/SPARK-48278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48278. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46585 [https://github.com/apache/spark/pull/46585] > Refine the string representation of `Cast` > -- > > Key: SPARK-48278 > URL: https://issues.apache.org/jira/browse/SPARK-48278 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48272) Add function `timestamp_diff`
[ https://issues.apache.org/jira/browse/SPARK-48272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-48272: - Assignee: Ruifeng Zheng > Add function `timestamp_diff` > - > > Key: SPARK-48272 > URL: https://issues.apache.org/jira/browse/SPARK-48272 > Project: Spark > Issue Type: New Feature > Components: Connect, PySpark, SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48272) Add function `timestamp_diff`
[ https://issues.apache.org/jira/browse/SPARK-48272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48272. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46576 [https://github.com/apache/spark/pull/46576] > Add function `timestamp_diff` > - > > Key: SPARK-48272 > URL: https://issues.apache.org/jira/browse/SPARK-48272 > Project: Spark > Issue Type: New Feature > Components: Connect, PySpark, SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48276) Add the missing __repr__ method for SQLExpression
[ https://issues.apache.org/jira/browse/SPARK-48276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-48276: -- Fix Version/s: 4.0.0 > Add the missing __repr__ method for SQLExpression > - > > Key: SPARK-48276 > URL: https://issues.apache.org/jira/browse/SPARK-48276 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48278) Refine the string representation of `Cast`
Ruifeng Zheng created SPARK-48278: - Summary: Refine the string representation of `Cast` Key: SPARK-48278 URL: https://issues.apache.org/jira/browse/SPARK-48278 Project: Spark Issue Type: Improvement Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48272) Add function `timestamp_diff`
Ruifeng Zheng created SPARK-48272: - Summary: Add function `timestamp_diff` Key: SPARK-48272 URL: https://issues.apache.org/jira/browse/SPARK-48272 Project: Spark Issue Type: New Feature Components: Connect, PySpark, SQL Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48259) Add 3 missing methods in dsl
[ https://issues.apache.org/jira/browse/SPARK-48259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48259. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46559 [https://github.com/apache/spark/pull/46559] > Add 3 missing methods in dsl > > > Key: SPARK-48259 > URL: https://issues.apache.org/jira/browse/SPARK-48259 > Project: Spark > Issue Type: Test > Components: Connect, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48259) Add 3 missing methods in dsl
Ruifeng Zheng created SPARK-48259: - Summary: Add 3 missing methods in dsl Key: SPARK-48259 URL: https://issues.apache.org/jira/browse/SPARK-48259 Project: Spark Issue Type: Test Components: Connect, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48228) Implement the missing function validation in ApplyInXXX
Ruifeng Zheng created SPARK-48228: - Summary: Implement the missing function validation in ApplyInXXX Key: SPARK-48228 URL: https://issues.apache.org/jira/browse/SPARK-48228 Project: Spark Issue Type: Sub-task Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48227) Document the requirement of seed in protos
Ruifeng Zheng created SPARK-48227: - Summary: Document the requirement of seed in protos Key: SPARK-48227 URL: https://issues.apache.org/jira/browse/SPARK-48227 Project: Spark Issue Type: Improvement Components: Documentation, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48190) Introduce a helper function to drop metadata
[ https://issues.apache.org/jira/browse/SPARK-48190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48190. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46466 [https://github.com/apache/spark/pull/46466] > Introduce a helper function to drop metadata > > > Key: SPARK-48190 > URL: https://issues.apache.org/jira/browse/SPARK-48190 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48184) Always set the seed of dataframe.sample in Client side
[
https://issues.apache.org/jira/browse/SPARK-48184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-48184:
--
Description:
In Spark Classic:
In [1]: df = spark.range(1).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006]{\{ }}
In Spark Connect:
In [1]: df = spark.range(1).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979]{{}}
was:
In Spark Classic:
In [1]: df = spark.range(1).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006]{{ }}
In Spark Connect:
In [1]: df = spark.range(1).sample(0.1)
In [2]: [df.count() for i in range(10)]
Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979]{{}}
> Always set the seed of dataframe.sample in Client side
> --
>
> Key: SPARK-48184
> URL: https://issues.apache.org/jira/browse/SPARK-48184
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
>Affects Versions: 4.0.0, 3.5.1, 3.4.3
>Reporter: Ruifeng Zheng
>Priority: Major
> Labels: pull-request-available
>
> In Spark Classic:
> In [1]: df = spark.range(1).sample(0.1)
> In [2]: [df.count() for i in range(10)]
> Out[2]: [1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006, 1006]{\{ }}
> In Spark Connect:
> In [1]: df = spark.range(1).sample(0.1)
> In [2]: [df.count() for i in range(10)]
> Out[2]: [969, 1005, 958, 996, 987, 1026, 991, 1020, 1012, 979]{{}}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-48142) Enable `CogroupedApplyInPandasTests.test_wrong_args`
[ https://issues.apache.org/jira/browse/SPARK-48142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-48142: - Assignee: Ruifeng Zheng > Enable `CogroupedApplyInPandasTests.test_wrong_args` > > > Key: SPARK-48142 > URL: https://issues.apache.org/jira/browse/SPARK-48142 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48142) Enable `CogroupedApplyInPandasTests.test_wrong_args`
[ https://issues.apache.org/jira/browse/SPARK-48142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48142. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46397 [https://github.com/apache/spark/pull/46397] > Enable `CogroupedApplyInPandasTests.test_wrong_args` > > > Key: SPARK-48142 > URL: https://issues.apache.org/jira/browse/SPARK-48142 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48142) Enable `CogroupedApplyInPandasTests.test_wrong_args`
Ruifeng Zheng created SPARK-48142: - Summary: Enable `CogroupedApplyInPandasTests.test_wrong_args` Key: SPARK-48142 URL: https://issues.apache.org/jira/browse/SPARK-48142 Project: Spark Issue Type: Sub-task Components: Connect, PySpark, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48058) `UserDefinedFunction.returnType` parse the DDL string
Ruifeng Zheng created SPARK-48058: - Summary: `UserDefinedFunction.returnType` parse the DDL string Key: SPARK-48058 URL: https://issues.apache.org/jira/browse/SPARK-48058 Project: Spark Issue Type: Sub-task Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-48055) Enable PandasUDFScalarParityTests.{test_vectorized_udf_empty_partition, test_vectorized_udf_struct_with_empty_partition}
[
https://issues.apache.org/jira/browse/SPARK-48055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng resolved SPARK-48055.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 46296
[https://github.com/apache/spark/pull/46296]
> Enable PandasUDFScalarParityTests.{test_vectorized_udf_empty_partition,
> test_vectorized_udf_struct_with_empty_partition}
>
>
> Key: SPARK-48055
> URL: https://issues.apache.org/jira/browse/SPARK-48055
> Project: Spark
> Issue Type: Sub-task
> Components: Connect, PySpark, Tests
>Affects Versions: 4.0.0
>Reporter: Ruifeng Zheng
>Assignee: Ruifeng Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47129) Make ResolveRelations cache connect plan properly
[ https://issues.apache.org/jira/browse/SPARK-47129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47129: -- Issue Type: Bug (was: Improvement) > Make ResolveRelations cache connect plan properly > - > > Key: SPARK-47129 > URL: https://issues.apache.org/jira/browse/SPARK-47129 > Project: Spark > Issue Type: Bug > Components: Connect, SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47129) Make ResolveRelations cache connect plan properly
[ https://issues.apache.org/jira/browse/SPARK-47129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47129: -- Affects Version/s: 3.4.3 3.5.1 > Make ResolveRelations cache connect plan properly > - > > Key: SPARK-47129 > URL: https://issues.apache.org/jira/browse/SPARK-47129 > Project: Spark > Issue Type: Bug > Components: Connect, SQL >Affects Versions: 4.0.0, 3.5.1, 3.4.3 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48020) Pin 'pandas==2.2.2'
Ruifeng Zheng created SPARK-48020: - Summary: Pin 'pandas==2.2.2' Key: SPARK-48020 URL: https://issues.apache.org/jira/browse/SPARK-48020 Project: Spark Issue Type: Bug Components: Project Infra, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48005) Enable `DefaultIndexParityTests. test_index_distributed_sequence_cleanup`
Ruifeng Zheng created SPARK-48005: - Summary: Enable `DefaultIndexParityTests. test_index_distributed_sequence_cleanup` Key: SPARK-48005 URL: https://issues.apache.org/jira/browse/SPARK-48005 Project: Spark Issue Type: Sub-task Components: Connect, PS Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47986) [CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server
[ https://issues.apache.org/jira/browse/SPARK-47986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47986. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46221 [https://github.com/apache/spark/pull/46221] > [CONNECT][PYTHON] Unable to create a new session when the default session is > closed by the server > - > > Key: SPARK-47986 > URL: https://issues.apache.org/jira/browse/SPARK-47986 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 3.5.0, 3.5.1 >Reporter: Niranjan Jayakar >Assignee: Niranjan Jayakar >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > When the server closes a session, usually after a cluster restart, the client > is unaware of this until it receives an error. > Once it does so, there is no way for the client to create a new session since > the stale sessions are still recorded as default and active sessions. > The only solution currently is to restart the Python interpreter on the > client, or to reach into the session builder and change the active or default > session. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47985) Simplify functions with `lit`
[ https://issues.apache.org/jira/browse/SPARK-47985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47985. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46219 [https://github.com/apache/spark/pull/46219] > Simplify functions with `lit` > - > > Key: SPARK-47985 > URL: https://issues.apache.org/jira/browse/SPARK-47985 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47985) Simplify functions with `lit`
[ https://issues.apache.org/jira/browse/SPARK-47985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47985: - Assignee: Ruifeng Zheng > Simplify functions with `lit` > - > > Key: SPARK-47985 > URL: https://issues.apache.org/jira/browse/SPARK-47985 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47623) Enable `QuietTest` in parity tests
[ https://issues.apache.org/jira/browse/SPARK-47623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47623: -- Summary: Enable `QuietTest` in parity tests (was: Use `QuietTest` in parity tests) > Enable `QuietTest` in parity tests > -- > > Key: SPARK-47623 > URL: https://issues.apache.org/jira/browse/SPARK-47623 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47987) Reenable `ArrowParityTests.test_createDataFrame_empty_partition`
Ruifeng Zheng created SPARK-47987: - Summary: Reenable `ArrowParityTests.test_createDataFrame_empty_partition` Key: SPARK-47987 URL: https://issues.apache.org/jira/browse/SPARK-47987 Project: Spark Issue Type: Sub-task Components: Connect, PySpark, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47970) Revisit skipped parity tests for PySpark Connect
[ https://issues.apache.org/jira/browse/SPARK-47970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47970: -- Summary: Revisit skipped parity tests for PySpark Connect (was: Revisit skipped parity tests for PySpark) > Revisit skipped parity tests for PySpark Connect > > > Key: SPARK-47970 > URL: https://issues.apache.org/jira/browse/SPARK-47970 > Project: Spark > Issue Type: Umbrella > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47985) Simplify functions with `lit`
Ruifeng Zheng created SPARK-47985: - Summary: Simplify functions with `lit` Key: SPARK-47985 URL: https://issues.apache.org/jira/browse/SPARK-47985 Project: Spark Issue Type: Improvement Components: PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47499) Reuse `test_help_command` in Connect
[ https://issues.apache.org/jira/browse/SPARK-47499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47499: -- Parent: SPARK-47970 Issue Type: Sub-task (was: Test) > Reuse `test_help_command` in Connect > > > Key: SPARK-47499 > URL: https://issues.apache.org/jira/browse/SPARK-47499 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47970) Revisit skipped parity tests for PySpark
Ruifeng Zheng created SPARK-47970: - Summary: Revisit skipped parity tests for PySpark Key: SPARK-47970 URL: https://issues.apache.org/jira/browse/SPARK-47970 Project: Spark Issue Type: Umbrella Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47937) Fix docstring of `hll_sketch_agg`
Ruifeng Zheng created SPARK-47937: - Summary: Fix docstring of `hll_sketch_agg` Key: SPARK-47937 URL: https://issues.apache.org/jira/browse/SPARK-47937 Project: Spark Issue Type: Improvement Components: Documentation, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47935) Pin pandas==2.0.3 for pypy3.8
Ruifeng Zheng created SPARK-47935: - Summary: Pin pandas==2.0.3 for pypy3.8 Key: SPARK-47935 URL: https://issues.apache.org/jira/browse/SPARK-47935 Project: Spark Issue Type: Improvement Components: Project Infra, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47845) Support column type in split function in scala and python
[
https://issues.apache.org/jira/browse/SPARK-47845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng resolved SPARK-47845.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 46045
[https://github.com/apache/spark/pull/46045]
> Support column type in split function in scala and python
> -
>
> Key: SPARK-47845
> URL: https://issues.apache.org/jira/browse/SPARK-47845
> Project: Spark
> Issue Type: New Feature
> Components: Connect, Spark Core
>Affects Versions: 3.5.1
>Reporter: Liu Cao
>Assignee: Liu Cao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> I have a use case to split a String typed column with different delimiters
> defined in other columns of the dataframe. SQL already supports this, but
> scala / python functions currently don't.
>
> A hypothetical example to illustrate:
> {code:java}
> import org.apache.spark.sql.functions.{col, split}
> val example = spark.createDataFrame(
> Seq(
> ("Doe, John", ", ", 2),
> ("Smith,Jane", ",", 2),
> ("Johnson", ",", 1)
> )
> )
> .toDF("name", "delim", "expected_parts_count")
> example.createOrReplaceTempView("test_data")
> // works for SQL
> spark.sql("SELECT split(name, delim, expected_parts_count) AS name_parts FROM
> test_data").show()
> // currently doesn't compile for scala, but easy to support
> example.withColumn("name_parts", split(col("name"), col("delim"),
> col("expected_parts_count"))).show() {code}
>
> Pretty simple patch that I can make a PR soon
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47845) Support column type in split function in scala and python
[
https://issues.apache.org/jira/browse/SPARK-47845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng reassigned SPARK-47845:
-
Assignee: Liu Cao
> Support column type in split function in scala and python
> -
>
> Key: SPARK-47845
> URL: https://issues.apache.org/jira/browse/SPARK-47845
> Project: Spark
> Issue Type: New Feature
> Components: Connect, Spark Core
>Affects Versions: 3.5.1
>Reporter: Liu Cao
>Assignee: Liu Cao
>Priority: Major
> Labels: pull-request-available
>
> I have a use case to split a String typed column with different delimiters
> defined in other columns of the dataframe. SQL already supports this, but
> scala / python functions currently don't.
>
> A hypothetical example to illustrate:
> {code:java}
> import org.apache.spark.sql.functions.{col, split}
> val example = spark.createDataFrame(
> Seq(
> ("Doe, John", ", ", 2),
> ("Smith,Jane", ",", 2),
> ("Johnson", ",", 1)
> )
> )
> .toDF("name", "delim", "expected_parts_count")
> example.createOrReplaceTempView("test_data")
> // works for SQL
> spark.sql("SELECT split(name, delim, expected_parts_count) AS name_parts FROM
> test_data").show()
> // currently doesn't compile for scala, but easy to support
> example.withColumn("name_parts", split(col("name"), col("delim"),
> col("expected_parts_count"))).show() {code}
>
> Pretty simple patch that I can make a PR soon
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47906) Fix docstring and type hint of `hll_union_agg`
[ https://issues.apache.org/jira/browse/SPARK-47906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47906. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46128 [https://github.com/apache/spark/pull/46128] > Fix docstring and type hint of `hll_union_agg` > -- > > Key: SPARK-47906 > URL: https://issues.apache.org/jira/browse/SPARK-47906 > Project: Spark > Issue Type: Improvement > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47883) Make CollectTailExec execute lazily
[ https://issues.apache.org/jira/browse/SPARK-47883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47883: - Assignee: Ruifeng Zheng > Make CollectTailExec execute lazily > > > Key: SPARK-47883 > URL: https://issues.apache.org/jira/browse/SPARK-47883 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47883) Make CollectTailExec execute lazily
[ https://issues.apache.org/jira/browse/SPARK-47883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47883. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46101 [https://github.com/apache/spark/pull/46101] > Make CollectTailExec execute lazily > > > Key: SPARK-47883 > URL: https://issues.apache.org/jira/browse/SPARK-47883 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47883) Make CollectTailExec execute lazily
[ https://issues.apache.org/jira/browse/SPARK-47883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47883: -- Summary: Make CollectTailExec execute lazily (was: Make CollectTailExec lazily execute) > Make CollectTailExec execute lazily > > > Key: SPARK-47883 > URL: https://issues.apache.org/jira/browse/SPARK-47883 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47883) Make CollectTailExec lazily execute
Ruifeng Zheng created SPARK-47883: - Summary: Make CollectTailExec lazily execute Key: SPARK-47883 URL: https://issues.apache.org/jira/browse/SPARK-47883 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47868) Recursion Limit Error in SparkSession and SparkConnectPlanner
[ https://issues.apache.org/jira/browse/SPARK-47868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47868. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46075 [https://github.com/apache/spark/pull/46075] > Recursion Limit Error in SparkSession and SparkConnectPlanner > - > > Key: SPARK-47868 > URL: https://issues.apache.org/jira/browse/SPARK-47868 > Project: Spark > Issue Type: Bug > Components: Connect >Affects Versions: 4.0.0 >Reporter: Tom van Bussel >Assignee: Tom van Bussel >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47855) Warn `spark.sql.execution.arrow.pyspark.fallback.enabled` in Connect
Ruifeng Zheng created SPARK-47855: - Summary: Warn `spark.sql.execution.arrow.pyspark.fallback.enabled` in Connect Key: SPARK-47855 URL: https://issues.apache.org/jira/browse/SPARK-47855 Project: Spark Issue Type: Improvement Components: Connect Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47828. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46023 [https://github.com/apache/spark/pull/46023] > DataFrameWriterV2.overwrite fails with invalid plan > --- > > Key: SPARK-47828 > URL: https://issues.apache.org/jira/browse/SPARK-47828 > Project: Spark > Issue Type: Bug > Components: Connect >Affects Versions: 3.4.2, 4.0.0, 3.5.1 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47828: - Assignee: Ruifeng Zheng > DataFrameWriterV2.overwrite fails with invalid plan > --- > > Key: SPARK-47828 > URL: https://issues.apache.org/jira/browse/SPARK-47828 > Project: Spark > Issue Type: Bug > Components: Connect >Affects Versions: 3.4.2, 4.0.0, 3.5.1 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47816) Document the lazy evaluation of views in spark.{sql, table}
[
https://issues.apache.org/jira/browse/SPARK-47816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng reassigned SPARK-47816:
-
Assignee: Ruifeng Zheng
> Document the lazy evaluation of views in spark.{sql, table}
> ---
>
> Key: SPARK-47816
> URL: https://issues.apache.org/jira/browse/SPARK-47816
> Project: Spark
> Issue Type: Improvement
> Components: Connect, Documentation
>Affects Versions: 4.0.0
>Reporter: Ruifeng Zheng
>Assignee: Ruifeng Zheng
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47816) Document the lazy evaluation of views in spark.{sql, table}
[
https://issues.apache.org/jira/browse/SPARK-47816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng resolved SPARK-47816.
---
Fix Version/s: 4.0.0
Resolution: Fixed
Issue resolved by pull request 46007
[https://github.com/apache/spark/pull/46007]
> Document the lazy evaluation of views in spark.{sql, table}
> ---
>
> Key: SPARK-47816
> URL: https://issues.apache.org/jira/browse/SPARK-47816
> Project: Spark
> Issue Type: Improvement
> Components: Connect, Documentation
>Affects Versions: 4.0.0
>Reporter: Ruifeng Zheng
>Assignee: Ruifeng Zheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47828: -- Affects Version/s: 3.4.2 > DataFrameWriterV2.overwrite fails with invalid plan > --- > > Key: SPARK-47828 > URL: https://issues.apache.org/jira/browse/SPARK-47828 > Project: Spark > Issue Type: Bug > Components: Connect >Affects Versions: 3.4.2, 4.0.0, 3.5.1 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan
[ https://issues.apache.org/jira/browse/SPARK-47828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47828: -- Issue Type: Bug (was: Improvement) > DataFrameWriterV2.overwrite fails with invalid plan > --- > > Key: SPARK-47828 > URL: https://issues.apache.org/jira/browse/SPARK-47828 > Project: Spark > Issue Type: Bug > Components: Connect >Affects Versions: 4.0.0, 3.5.1 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47828) DataFrameWriterV2.overwrite fails with invalid plan
Ruifeng Zheng created SPARK-47828: - Summary: DataFrameWriterV2.overwrite fails with invalid plan Key: SPARK-47828 URL: https://issues.apache.org/jira/browse/SPARK-47828 Project: Spark Issue Type: Improvement Components: Connect Affects Versions: 3.5.1, 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47815) Unify the user agent with json
[ https://issues.apache.org/jira/browse/SPARK-47815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47815. --- Resolution: Not A Problem > Unify the user agent with json > -- > > Key: SPARK-47815 > URL: https://issues.apache.org/jira/browse/SPARK-47815 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47816) Document the lazy evaluation of views in spark.{sql, table}
Ruifeng Zheng created SPARK-47816:
-
Summary: Document the lazy evaluation of views in spark.{sql,
table}
Key: SPARK-47816
URL: https://issues.apache.org/jira/browse/SPARK-47816
Project: Spark
Issue Type: Improvement
Components: Connect, Documentation
Affects Versions: 4.0.0
Reporter: Ruifeng Zheng
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47815) Unify the user agent string with json
[ https://issues.apache.org/jira/browse/SPARK-47815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47815: -- Summary: Unify the user agent string with json (was: Unify the user agent string representation with json) > Unify the user agent string with json > - > > Key: SPARK-47815 > URL: https://issues.apache.org/jira/browse/SPARK-47815 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47815) Unify the user agent string representation with json
Ruifeng Zheng created SPARK-47815: - Summary: Unify the user agent string representation with json Key: SPARK-47815 URL: https://issues.apache.org/jira/browse/SPARK-47815 Project: Spark Issue Type: Improvement Components: Connect Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47815) Unify the user agent with json
[ https://issues.apache.org/jira/browse/SPARK-47815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47815: -- Summary: Unify the user agent with json (was: Unify the user agent string with json) > Unify the user agent with json > -- > > Key: SPARK-47815 > URL: https://issues.apache.org/jira/browse/SPARK-47815 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47779) Add a helper function to sort PS Frame/Series
[ https://issues.apache.org/jira/browse/SPARK-47779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47779: - Assignee: Ruifeng Zheng > Add a helper function to sort PS Frame/Series > - > > Key: SPARK-47779 > URL: https://issues.apache.org/jira/browse/SPARK-47779 > Project: Spark > Issue Type: Improvement > Components: PS, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47779) Add a helper function to sort PS Frame/Series
[ https://issues.apache.org/jira/browse/SPARK-47779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47779. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45952 [https://github.com/apache/spark/pull/45952] > Add a helper function to sort PS Frame/Series > - > > Key: SPARK-47779 > URL: https://issues.apache.org/jira/browse/SPARK-47779 > Project: Spark > Issue Type: Improvement > Components: PS, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47779) Add a helper function to sort PS Frame/Series
Ruifeng Zheng created SPARK-47779: - Summary: Add a helper function to sort PS Frame/Series Key: SPARK-47779 URL: https://issues.apache.org/jira/browse/SPARK-47779 Project: Spark Issue Type: Improvement Components: PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47779) Add a helper function to sort PS Frame/Series
[ https://issues.apache.org/jira/browse/SPARK-47779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47779: -- Component/s: PS Tests (was: PySpark) > Add a helper function to sort PS Frame/Series > - > > Key: SPARK-47779 > URL: https://issues.apache.org/jira/browse/SPARK-47779 > Project: Spark > Issue Type: Improvement > Components: PS, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47772) Fix the doctest of mode function
[ https://issues.apache.org/jira/browse/SPARK-47772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47772. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45940 [https://github.com/apache/spark/pull/45940] > Fix the doctest of mode function > > > Key: SPARK-47772 > URL: https://issues.apache.org/jira/browse/SPARK-47772 > Project: Spark > Issue Type: Improvement > Components: PySpark, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47771) Make max_by, min_by doctests deterministic
[ https://issues.apache.org/jira/browse/SPARK-47771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47771. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45939 [https://github.com/apache/spark/pull/45939] > Make max_by, min_by doctests deterministic > -- > > Key: SPARK-47771 > URL: https://issues.apache.org/jira/browse/SPARK-47771 > Project: Spark > Issue Type: Improvement > Components: PySpark, Tests >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47771) Make max_by, min_by doctests deterministic
Ruifeng Zheng created SPARK-47771: - Summary: Make max_by, min_by doctests deterministic Key: SPARK-47771 URL: https://issues.apache.org/jira/browse/SPARK-47771 Project: Spark Issue Type: Improvement Components: PySpark, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47749) Dataframe.collect should accept duplicated column names
Ruifeng Zheng created SPARK-47749:
-
Summary: Dataframe.collect should accept duplicated column names
Key: SPARK-47749
URL: https://issues.apache.org/jira/browse/SPARK-47749
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.0.0
Reporter: Ruifeng Zheng
{code:java}
+---+---+---+---+
| i| j| i| j|
+---+---+---+---+
| 1| a| 1| a|
+---+---+---+---+ {code}
collect fails with
{code:java}
[info] org.apache.spark.sql.AnalysisException: [AMBIGUOUS_COLUMN_OR_FIELD]
Column or field `i` is ambiguous and has 2 matches. SQLSTATE: 42702
[info] at
org.apache.spark.sql.errors.CompilationErrors.ambiguousColumnOrFieldError(CompilationErrors.scala:28)
[info] at
org.apache.spark.sql.errors.CompilationErrors.ambiguousColumnOrFieldError$(CompilationErrors.scala:23)
[info] at
org.apache.spark.sql.errors.CompilationErrors$.ambiguousColumnOrFieldError(CompilationErrors.scala:54)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.$anonfun$createFieldLookup$1(ArrowDeserializer.scala:460)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.$anonfun$createFieldLookup$1$adapted(ArrowDeserializer.scala:454)
[info] at scala.collection.immutable.List.foreach(List.scala:334)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.createFieldLookup(ArrowDeserializer.scala:454)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:328)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializers$.deserializerFor(ArrowDeserializer.scala:86)
[info] at
org.apache.spark.sql.connect.client.arrow.ArrowDeserializingIterator.(ArrowDeserializer.scala:542)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47669) Add `try_cast` function in DataFrame
[ https://issues.apache.org/jira/browse/SPARK-47669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47669. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45796 [https://github.com/apache/spark/pull/45796] > Add `try_cast` function in DataFrame > > > Key: SPARK-47669 > URL: https://issues.apache.org/jira/browse/SPARK-47669 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark, SQL >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47664) Validate the column name with cached schema
Ruifeng Zheng created SPARK-47664: - Summary: Validate the column name with cached schema Key: SPARK-47664 URL: https://issues.apache.org/jira/browse/SPARK-47664 Project: Spark Issue Type: Improvement Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47638) Skip column name validation in PS
[ https://issues.apache.org/jira/browse/SPARK-47638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47638. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45752 [https://github.com/apache/spark/pull/45752] > Skip column name validation in PS > - > > Key: SPARK-47638 > URL: https://issues.apache.org/jira/browse/SPARK-47638 > Project: Spark > Issue Type: Improvement > Components: Connect, PS >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47638) Skip column name validation in PS
[ https://issues.apache.org/jira/browse/SPARK-47638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47638: - Assignee: Ruifeng Zheng > Skip column name validation in PS > - > > Key: SPARK-47638 > URL: https://issues.apache.org/jira/browse/SPARK-47638 > Project: Spark > Issue Type: Improvement > Components: Connect, PS >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47638) Skip column name validation in PS
Ruifeng Zheng created SPARK-47638: - Summary: Skip column name validation in PS Key: SPARK-47638 URL: https://issues.apache.org/jira/browse/SPARK-47638 Project: Spark Issue Type: Improvement Components: Connect, PS Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47621) Refine docstring of `try_sum`, `try_avg`, `avg`, `sum`, `mean`
[ https://issues.apache.org/jira/browse/SPARK-47621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47621. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45745 [https://github.com/apache/spark/pull/45745] > Refine docstring of `try_sum`, `try_avg`, `avg`, `sum`, `mean` > -- > > Key: SPARK-47621 > URL: https://issues.apache.org/jira/browse/SPARK-47621 > Project: Spark > Issue Type: Sub-task > Components: Documentation, PySpark >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47562) Factor literal handling out of `plan.py`
[ https://issues.apache.org/jira/browse/SPARK-47562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47562. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45719 [https://github.com/apache/spark/pull/45719] > Factor literal handling out of `plan.py` > > > Key: SPARK-47562 > URL: https://issues.apache.org/jira/browse/SPARK-47562 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47562) Factor literal handling out of `plan.py`
[ https://issues.apache.org/jira/browse/SPARK-47562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47562: - Assignee: Ruifeng Zheng > Factor literal handling out of `plan.py` > > > Key: SPARK-47562 > URL: https://issues.apache.org/jira/browse/SPARK-47562 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47562) Factor literal handling out of `plan.py`
Ruifeng Zheng created SPARK-47562: - Summary: Factor literal handling out of `plan.py` Key: SPARK-47562 URL: https://issues.apache.org/jira/browse/SPARK-47562 Project: Spark Issue Type: Improvement Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47560) Avoid RPC to validate column name with cached schema
[ https://issues.apache.org/jira/browse/SPARK-47560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47560. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45717 [https://github.com/apache/spark/pull/45717] > Avoid RPC to validate column name with cached schema > > > Key: SPARK-47560 > URL: https://issues.apache.org/jira/browse/SPARK-47560 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47560) Avoid RPC to validate column name with cached schema
[ https://issues.apache.org/jira/browse/SPARK-47560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47560: - Assignee: Ruifeng Zheng > Avoid RPC to validate column name with cached schema > > > Key: SPARK-47560 > URL: https://issues.apache.org/jira/browse/SPARK-47560 > Project: Spark > Issue Type: Improvement > Components: Connect, PySpark >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47560) Avoid RPC to validate column name with cached schema
Ruifeng Zheng created SPARK-47560: - Summary: Avoid RPC to validate column name with cached schema Key: SPARK-47560 URL: https://issues.apache.org/jira/browse/SPARK-47560 Project: Spark Issue Type: Improvement Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47500) Factor column name handling out of `plan.py`
Ruifeng Zheng created SPARK-47500: - Summary: Factor column name handling out of `plan.py` Key: SPARK-47500 URL: https://issues.apache.org/jira/browse/SPARK-47500 Project: Spark Issue Type: Improvement Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47499) Reuse `test_help_command` in Connect
Ruifeng Zheng created SPARK-47499: - Summary: Reuse `test_help_command` in Connect Key: SPARK-47499 URL: https://issues.apache.org/jira/browse/SPARK-47499 Project: Spark Issue Type: Test Components: Connect, PySpark, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-47436) Fix docstring links and type hints in Python Data Source
[ https://issues.apache.org/jira/browse/SPARK-47436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47436: - Assignee: Hyukjin Kwon > Fix docstring links and type hints in Python Data Source > > > Key: SPARK-47436 > URL: https://issues.apache.org/jira/browse/SPARK-47436 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-47436) Fix docstring links and type hints in Python Data Source
[ https://issues.apache.org/jira/browse/SPARK-47436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47436. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45557 [https://github.com/apache/spark/pull/45557] > Fix docstring links and type hints in Python Data Source > > > Key: SPARK-47436 > URL: https://issues.apache.org/jira/browse/SPARK-47436 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 4.0.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47437) Correct the error class for `DataFrame.sort`
Ruifeng Zheng created SPARK-47437: - Summary: Correct the error class for `DataFrame.sort` Key: SPARK-47437 URL: https://issues.apache.org/jira/browse/SPARK-47437 Project: Spark Issue Type: Bug Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41762) Make `Column.__neg__` return the same column name as PySpark
[ https://issues.apache.org/jira/browse/SPARK-41762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-41762: - Assignee: Ruifeng Zheng > Make `Column.__neg__` return the same column name as PySpark > > > Key: SPARK-41762 > URL: https://issues.apache.org/jira/browse/SPARK-41762 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > > [left]: Index(['negative(a)'], dtype='object') > [right]: Index(['(- a)'], dtype='object') -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-41762) Make `Column.__neg__` return the same column name as PySpark
[ https://issues.apache.org/jira/browse/SPARK-41762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-41762. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45493 [https://github.com/apache/spark/pull/45493] > Make `Column.__neg__` return the same column name as PySpark > > > Key: SPARK-41762 > URL: https://issues.apache.org/jira/browse/SPARK-41762 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > [left]: Index(['negative(a)'], dtype='object') > [right]: Index(['(- a)'], dtype='object') -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47377) Factor out tests from `SparkConnectSQLTestCase`
Ruifeng Zheng created SPARK-47377: - Summary: Factor out tests from `SparkConnectSQLTestCase` Key: SPARK-47377 URL: https://issues.apache.org/jira/browse/SPARK-47377 Project: Spark Issue Type: Sub-task Components: Connect, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47347) Factor session-related tests out of test_connect_basic
Ruifeng Zheng created SPARK-47347: - Summary: Factor session-related tests out of test_connect_basic Key: SPARK-47347 URL: https://issues.apache.org/jira/browse/SPARK-47347 Project: Spark Issue Type: Sub-task Components: Connect, PySpark, Tests Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47334) Make `withColumnRenamed` reuse the implementation of `withColumnsRenamed`
Ruifeng Zheng created SPARK-47334: - Summary: Make `withColumnRenamed` reuse the implementation of `withColumnsRenamed` Key: SPARK-47334 URL: https://issues.apache.org/jira/browse/SPARK-47334 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47322) Make `withColumnsRenamed` duplicated column name handling consisten with `withColumnRenamed`
Ruifeng Zheng created SPARK-47322: - Summary: Make `withColumnsRenamed` duplicated column name handling consisten with `withColumnRenamed` Key: SPARK-47322 URL: https://issues.apache.org/jira/browse/SPARK-47322 Project: Spark Issue Type: Improvement Components: Connect, PySpark Affects Versions: 4.0.0 Reporter: Ruifeng Zheng -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-46988) proto message abbreviation should support map fields
[ https://issues.apache.org/jira/browse/SPARK-46988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-46988: - Assignee: Ruifeng Zheng > proto message abbreviation should support map fields > > > Key: SPARK-46988 > URL: https://issues.apache.org/jira/browse/SPARK-46988 > Project: Spark > Issue Type: Improvement > Components: Connect >Affects Versions: 4.0.0 >Reporter: Ruifeng Zheng >Assignee: Ruifeng Zheng >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
