[jira] [Resolved] (SPARK-47978) Decouple Spark Go Connect Library versioning from Spark versioning

[ https://issues.apache.org/jira/browse/SPARK-47978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BoYang resolved SPARK-47978. Resolution: Fixed > Decouple Spark Go Connect Library versioning from Spark versioning >

[jira] [Updated] (SPARK-45225) XML: XSD file URL support

[ https://issues.apache.org/jira/browse/SPARK-45225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45225: --- Labels: pull-request-available (was: ) > XML: XSD file URL support >

[jira] [Resolved] (SPARK-47991) Arrange the test cases for window frames and window functions.

[ https://issues.apache.org/jira/browse/SPARK-47991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47991. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46226

[jira] [Created] (SPARK-48000) Hash join support for strings with collation

Uroš Bojanić created SPARK-48000: Summary: Hash join support for strings with collation Key: SPARK-48000 URL: https://issues.apache.org/jira/browse/SPARK-48000 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22231) Support of map, filter, withField, dropFields in nested list of structures

[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840934#comment-17840934 ] Dongjoon Hyun commented on SPARK-22231: --- I removed the outdated target version from this issue. >

[jira] [Updated] (SPARK-22231) Support of map, filter, withField, dropFields in nested list of structures

[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22231: -- Target Version/s: (was: 3.2.0) > Support of map, filter, withField, dropFields in nested

[jira] [Updated] (SPARK-24941) Add RDDBarrier.coalesce() function

[ https://issues.apache.org/jira/browse/SPARK-24941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24941: -- Target Version/s: (was: 3.2.0) > Add RDDBarrier.coalesce() function >

[jira] [Updated] (SPARK-25383) Image data source supports sample pushdown

[ https://issues.apache.org/jira/browse/SPARK-25383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25383: -- Target Version/s: (was: 3.2.0) > Image data source supports sample pushdown >

[jira] [Updated] (SPARK-25752) Add trait to easily whitelist logical operators that produce named output from CleanupAliases

[ https://issues.apache.org/jira/browse/SPARK-25752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25752: -- Target Version/s: (was: 3.2.0) > Add trait to easily whitelist logical operators that

[jira] [Commented] (SPARK-28629) Capture the missing rules in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-28629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840928#comment-17840928 ] Dongjoon Hyun commented on SPARK-28629: --- I removed the outdated target version from this issue. >

[jira] [Commented] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

[ https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840930#comment-17840930 ] Dongjoon Hyun commented on SPARK-27780: --- I removed the outdated target version from this issue. >

[jira] [Updated] (SPARK-28629) Capture the missing rules in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-28629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28629: -- Target Version/s: (was: 3.2.0) > Capture the missing rules in HiveSessionStateBuilder >

[jira] [Updated] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

[ https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27780: -- Target Version/s: (was: 3.2.0) > Shuffle server & client should be versioned to enable

[jira] [Commented] (SPARK-30324) Simplify API for JSON access in DataFrames/SQL

[ https://issues.apache.org/jira/browse/SPARK-30324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840927#comment-17840927 ] Dongjoon Hyun commented on SPARK-30324: --- I removed the outdated target version from this issue. >

[jira] [Updated] (SPARK-30324) Simplify API for JSON access in DataFrames/SQL

[ https://issues.apache.org/jira/browse/SPARK-30324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30324: -- Target Version/s: (was: 3.2.0) > Simplify API for JSON access in DataFrames/SQL >

[jira] [Updated] (SPARK-30334) Add metadata around semi-structured columns to Spark

[ https://issues.apache.org/jira/browse/SPARK-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30334: -- Target Version/s: (was: 3.2.0) > Add metadata around semi-structured columns to Spark >

[jira] [Commented] (SPARK-30334) Add metadata around semi-structured columns to Spark

[ https://issues.apache.org/jira/browse/SPARK-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840926#comment-17840926 ] Dongjoon Hyun commented on SPARK-30334: --- I removed the outdated target version from this issue. >

[jira] [Commented] (SPARK-24942) Improve cluster resource management with jobs containing barrier stage

[ https://issues.apache.org/jira/browse/SPARK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840913#comment-17840913 ] Dongjoon Hyun commented on SPARK-24942: --- I removed the outdated target version, `3.2.0`, from this

[jira] [Created] (SPARK-47998) pandas-on-spark DataFrame.concat will not join a Pandas dataframe and raises a misleading error

Philip Kahn created SPARK-47998: --- Summary: pandas-on-spark DataFrame.concat will not join a Pandas dataframe and raises a misleading error Key: SPARK-47998 URL: https://issues.apache.org/jira/browse/SPARK-47998

[jira] [Updated] (SPARK-24942) Improve cluster resource management with jobs containing barrier stage

[ https://issues.apache.org/jira/browse/SPARK-24942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24942: -- Target Version/s: (was: 3.2.0) > Improve cluster resource management with jobs containing

[jira] [Created] (SPARK-47997) Pandas-on-Spark incompletely implements DataFrame.drop

Philip Kahn created SPARK-47997: --- Summary: Pandas-on-Spark incompletely implements DataFrame.drop Key: SPARK-47997 URL: https://issues.apache.org/jira/browse/SPARK-47997 Project: Spark Issue

[jira] [Created] (SPARK-47996) Pandas-on-Spark incompletely implements merge methods

Philip Kahn created SPARK-47996: --- Summary: Pandas-on-Spark incompletely implements merge methods Key: SPARK-47996 URL: https://issues.apache.org/jira/browse/SPARK-47996 Project: Spark Issue

[jira] [Created] (SPARK-47994) SQLServer does not support 1 and 0 as boolean values

Stefan Bukorovic created SPARK-47994: Summary: SQLServer does not support 1 and 0 as boolean values Key: SPARK-47994 URL: https://issues.apache.org/jira/browse/SPARK-47994 Project: Spark

[jira] [Commented] (SPARK-44111) Prepare Apache Spark 4.0.0

[ https://issues.apache.org/jira/browse/SPARK-44111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840853#comment-17840853 ] Dongjoon Hyun commented on SPARK-44111: --- Yes, we will provide `4.0.0-preview` in advance,

[jira] [Updated] (SPARK-47355) Use wildcard imports in CollationTypeCasts

[ https://issues.apache.org/jira/browse/SPARK-47355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47355: --- Labels: pull-request-available (was: ) > Use wildcard imports in CollationTypeCasts >

[jira] [Updated] (SPARK-47355) Use wildcard imports in CollationTypeCast

[ https://issues.apache.org/jira/browse/SPARK-47355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47355: - Summary: Use wildcard imports in CollationTypeCast (was: TBD) > Use wildcard imports in

[jira] [Updated] (SPARK-47355) Use wildcard imports in CollationTypeCasts

[ https://issues.apache.org/jira/browse/SPARK-47355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47355: - Summary: Use wildcard imports in CollationTypeCasts (was: Use wildcard imports in

[jira] [Resolved] (SPARK-47987) Enable `ArrowParityTests.test_createDataFrame_empty_partition`

[ https://issues.apache.org/jira/browse/SPARK-47987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47987. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46220

[jira] [Resolved] (SPARK-47990) Upgrade `zstd-jni` to 1.5.6-3

[ https://issues.apache.org/jira/browse/SPARK-47990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47990. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46225

[jira] [Commented] (SPARK-46830) Introducing collation concept into Spark

[ https://issues.apache.org/jira/browse/SPARK-46830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840792#comment-17840792 ] Gideon P commented on SPARK-46830: -- [~uros-db] what should I work on next? > Introducing collation

[jira] [Commented] (SPARK-44111) Prepare Apache Spark 4.0.0

[ https://issues.apache.org/jira/browse/SPARK-44111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840788#comment-17840788 ] Florent BIVILLE commented on SPARK-44111: - Is there going to be pre-releases for Spark 4 that

[jira] [Resolved] (SPARK-47985) Simplify functions with `lit`

[ https://issues.apache.org/jira/browse/SPARK-47985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-47985. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46219

[jira] [Assigned] (SPARK-47985) Simplify functions with `lit`

[ https://issues.apache.org/jira/browse/SPARK-47985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-47985: - Assignee: Ruifeng Zheng > Simplify functions with `lit` >

[jira] [Updated] (SPARK-47993) Drop Python 3.8 support

[ https://issues.apache.org/jira/browse/SPARK-47993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-47993: - Labels: release-notes (was: release-note) > Drop Python 3.8 support > ---

[jira] [Updated] (SPARK-47963) Make the external Spark ecosystem can use structured logging mechanisms

[ https://issues.apache.org/jira/browse/SPARK-47963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-47963: Summary: Make the external Spark ecosystem can use structured logging mechanisms (was: Add an

[jira] [Created] (SPARK-47992) Support recursive descent path in get_json_object function

Qian Sun created SPARK-47992: Summary: Support recursive descent path in get_json_object function Key: SPARK-47992 URL: https://issues.apache.org/jira/browse/SPARK-47992 Project: Spark Issue

[jira] [Updated] (SPARK-47991) Arrange the test cases for window frames and window functions.

[ https://issues.apache.org/jira/browse/SPARK-47991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47991: --- Labels: pull-request-available (was: ) > Arrange the test cases for window frames and

[jira] [Created] (SPARK-47991) Arrange the test cases for window frames and window functions.

Jiaan Geng created SPARK-47991: -- Summary: Arrange the test cases for window frames and window functions. Key: SPARK-47991 URL: https://issues.apache.org/jira/browse/SPARK-47991 Project: Spark

[jira] [Commented] (SPARK-38958) Override S3 Client in Spark Write/Read calls

[ https://issues.apache.org/jira/browse/SPARK-38958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840748#comment-17840748 ] ASF GitHub Bot commented on SPARK-38958: hadoop-yetus commented on PR #6550: URL:

[jira] [Updated] (SPARK-47297) TBD

[ https://issues.apache.org/jira/browse/SPARK-47297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47297: - Summary: TBD (was: split (binary & lowercase collation only)) > TBD > --- > >

[jira] [Updated] (SPARK-47408) Fix mathExpressions that use StringType

[ https://issues.apache.org/jira/browse/SPARK-47408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihailo Milosevic updated SPARK-47408: -- Summary: Fix mathExpressions that use StringType (was: TBD) > Fix mathExpressions

[jira] [Updated] (SPARK-47353) Mode (all collations)

[ https://issues.apache.org/jira/browse/SPARK-47353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47353: - Description: Enable collation support for the *Mode* expression in Spark. First confirm what

[jira] [Updated] (SPARK-47353) Mode (all collations)

[ https://issues.apache.org/jira/browse/SPARK-47353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47353: - Description: Enable collation support for the *Mode* expression in Spark. First confirm what

[jira] [Commented] (SPARK-47353) Mode (all collations)

[ https://issues.apache.org/jira/browse/SPARK-47353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840728#comment-17840728 ] Uroš Bojanić commented on SPARK-47353: -- [~panbingkun] if you're looking to make some contributions

[jira] [Assigned] (SPARK-47566) SubstringIndex

[ https://issues.apache.org/jira/browse/SPARK-47566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47566: -- Assignee: Apache Spark > SubstringIndex > -- > > Key:

[jira] [Assigned] (SPARK-47566) SubstringIndex

[ https://issues.apache.org/jira/browse/SPARK-47566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47566: -- Assignee: (was: Apache Spark) > SubstringIndex > -- > >

[jira] [Updated] (SPARK-47623) Enable `QuietTest` in parity tests

[ https://issues.apache.org/jira/browse/SPARK-47623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47623: -- Summary: Enable `QuietTest` in parity tests (was: Use `QuietTest` in parity tests) > Enable

[jira] [Resolved] (SPARK-47982) Update code style' plugins to latest version

[ https://issues.apache.org/jira/browse/SPARK-47982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-47982. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46216

[jira] [Updated] (SPARK-47986) [CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server

[ https://issues.apache.org/jira/browse/SPARK-47986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47986: --- Labels: pull-request-available (was: ) > [CONNECT][PYTHON] Unable to create a new session

[jira] [Created] (SPARK-47987) Reenable `ArrowParityTests.test_createDataFrame_empty_partition`

Ruifeng Zheng created SPARK-47987: - Summary: Reenable `ArrowParityTests.test_createDataFrame_empty_partition` Key: SPARK-47987 URL: https://issues.apache.org/jira/browse/SPARK-47987 Project: Spark

[jira] [Resolved] (SPARK-47984) Change `MetricsAggregate/V2Aggregator`'s `serialize/deserialize` to call `SparkSerDeUtils`'s `serialize/deserialize` methods.

[ https://issues.apache.org/jira/browse/SPARK-47984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-47984. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46218

[jira] [Assigned] (SPARK-47984) Change `MetricsAggregate/V2Aggregator`'s `serialize/deserialize` to call `SparkSerDeUtils`'s `serialize/deserialize` methods.

[ https://issues.apache.org/jira/browse/SPARK-47984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-47984: Assignee: Yang Jie > Change `MetricsAggregate/V2Aggregator`'s `serialize/deserialize` to call >

[jira] [Updated] (SPARK-47970) Revisit skipped parity tests for PySpark Connect

[ https://issues.apache.org/jira/browse/SPARK-47970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-47970: -- Summary: Revisit skipped parity tests for PySpark Connect (was: Revisit skipped parity tests

[jira] [Resolved] (SPARK-47983) Demote spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled to internal

[ https://issues.apache.org/jira/browse/SPARK-47983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-47983. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46217

[jira] [Updated] (SPARK-47985) Simplify functions with `lit`

[ https://issues.apache.org/jira/browse/SPARK-47985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47985: --- Labels: pull-request-available (was: ) > Simplify functions with `lit` >

[jira] [Created] (SPARK-47985) Simplify functions with `lit`

Ruifeng Zheng created SPARK-47985: - Summary: Simplify functions with `lit` Key: SPARK-47985 URL: https://issues.apache.org/jira/browse/SPARK-47985 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-47984) Change `MetricsAggregate/V2Aggregator`'s `serialize/deserialize` to call `SparkSerDeUtils`'s `serialize/deserialize` methods.

[ https://issues.apache.org/jira/browse/SPARK-47984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47984: --- Labels: pull-request-available (was: ) > Change `MetricsAggregate/V2Aggregator`'s

[jira] [Created] (SPARK-47984) Change `MetricsAggregate/V2Aggregator`'s `serialize/deserialize` to call `SparkSerDeUtils`'s `serialize/deserialize` methods.

Yang Jie created SPARK-47984: Summary: Change `MetricsAggregate/V2Aggregator`'s `serialize/deserialize` to call `SparkSerDeUtils`'s `serialize/deserialize` methods. Key: SPARK-47984 URL:

[jira] [Updated] (SPARK-47983) Demote spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled to internal

[ https://issues.apache.org/jira/browse/SPARK-47983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47983: --- Labels: pull-request-available (was: ) > Demote

[jira] [Created] (SPARK-47983) Demote spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled to internal

Kent Yao created SPARK-47983: Summary: Demote spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled to internal Key: SPARK-47983 URL: https://issues.apache.org/jira/browse/SPARK-47983 Project: