[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications

2023-01-03 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-41053: --- Labels: release-notes (was: ) > Better Spark UI scalability and Driver stability for large

[jira] [Commented] (SPARK-41833) DataFrame.collect() output parity with pyspark

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654327#comment-17654327 ] Apache Spark commented on SPARK-41833: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-41833) DataFrame.collect() output parity with pyspark

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41833: Assignee: Apache Spark > DataFrame.collect() output parity with pyspark >

[jira] [Assigned] (SPARK-41833) DataFrame.collect() output parity with pyspark

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41833: Assignee: (was: Apache Spark) > DataFrame.collect() output parity with pyspark >

[jira] [Commented] (SPARK-41833) DataFrame.collect() output parity with pyspark

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654326#comment-17654326 ] Apache Spark commented on SPARK-41833: -- User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-41432) Protobuf serializer for SparkPlanGraphWrapper

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654323#comment-17654323 ] Apache Spark commented on SPARK-41432: -- User 'LuciferYang' has created a pull request for this

[jira] [Updated] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-01-03 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Description: Python user-defined function (UDF) enables users to run arbitrary code against

[jira] [Commented] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654322#comment-17654322 ] Apache Spark commented on SPARK-40307: -- User 'xinrong-meng' has created a pull request for this

[jira] [Updated] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-01-03 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Description: Python user-defined function (UDF) enables users to run arbitrary code against

[jira] [Assigned] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40307: Assignee: Apache Spark > Introduce Arrow-optimized Python UDFs >

[jira] [Updated] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-01-03 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Description: Python user-defined function (UDF) enables users to run arbitrary code against

[jira] [Created] (SPARK-41881) `DataFrame.collect` should handle None/NaN properly

2023-01-03 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-41881: - Summary: `DataFrame.collect` should handle None/NaN properly Key: SPARK-41881 URL: https://issues.apache.org/jira/browse/SPARK-41881 Project: Spark Issue

[jira] [Assigned] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40307: Assignee: (was: Apache Spark) > Introduce Arrow-optimized Python UDFs >

[jira] [Updated] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-01-03 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Summary: Introduce Arrow-optimized Python UDFs (was: Optimize (De)Serialization of Python UDFs

[jira] [Updated] (SPARK-40307) Optimize (De)Serialization of Python UDFs by Arrow

2023-01-03 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Description: Python user-defined function (UDF) enables users to run arbitrary code against

[jira] [Updated] (SPARK-40307) Optimize (De)Serialization of Python UDFs by Arrow

2023-01-03 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Summary: Optimize (De)Serialization of Python UDFs by Arrow (was: Optimize (De)Serialization

[jira] [Created] (SPARK-41880) Function `from_json` should support non-literal expression

2023-01-03 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-41880: - Summary: Function `from_json` should support non-literal expression Key: SPARK-41880 URL: https://issues.apache.org/jira/browse/SPARK-41880 Project: Spark

[jira] [Created] (SPARK-41879) `DataFrame.collect` should support nested types

2023-01-03 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-41879: - Summary: `DataFrame.collect` should support nested types Key: SPARK-41879 URL: https://issues.apache.org/jira/browse/SPARK-41879 Project: Spark Issue

[jira] [Commented] (SPARK-41780) `regexp_replace('', '[a\\\\d]{0, 2}', 'x')` causes an internal error

2023-01-03 Thread Remzi Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654311#comment-17654311 ] Remzi Yang commented on SPARK-41780: It is a bug I guess, because an internal error is returned.

[jira] [Commented] (SPARK-41780) `regexp_replace('', '[a\\\\d]{0, 2}', 'x')` causes an internal error

2023-01-03 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654305#comment-17654305 ] BingKun Pan commented on SPARK-41780: - This is not bug, only the error prompt is not clear.

[jira] [Updated] (SPARK-41780) `regexp_replace('', '[a\\\\d]{0, 2}', 'x')` causes an internal error

2023-01-03 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-41780: Attachment: image-2023-01-04-14-12-26-126.png > `regexp_replace('', '[ad]{0, 2}', 'x')`

[jira] [Updated] (SPARK-39596) Run `Linters, licenses, dependencies and documentation generation ` GitHub Actions failed

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39596: -- Fix Version/s: 3.2.4 > Run `Linters, licenses, dependencies and documentation generation `

[jira] [Updated] (SPARK-38261) Sync missing R packages with CI

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38261: -- Fix Version/s: 3.2.4 > Sync missing R packages with CI > --- > >

[jira] [Assigned] (SPARK-41780) `regexp_replace('', '[a\\\\d]{0, 2}', 'x')` causes an internal error

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41780: Assignee: Apache Spark > `regexp_replace('', '[ad]{0, 2}', 'x')` causes an internal

[jira] [Commented] (SPARK-41780) `regexp_replace('', '[a\\\\d]{0, 2}', 'x')` causes an internal error

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654300#comment-17654300 ] Apache Spark commented on SPARK-41780: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-41780) `regexp_replace('', '[a\\\\d]{0, 2}', 'x')` causes an internal error

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41780: Assignee: (was: Apache Spark) > `regexp_replace('', '[ad]{0, 2}', 'x')` causes

[jira] [Assigned] (SPARK-41878) Add JIRAs or messages for skipped messages

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41878: Assignee: Hyukjin Kwon (was: Apache Spark) > Add JIRAs or messages for skipped messages

[jira] [Commented] (SPARK-41878) Add JIRAs or messages for skipped messages

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654297#comment-17654297 ] Apache Spark commented on SPARK-41878: -- User 'techaddict' has created a pull request for this

[jira] [Assigned] (SPARK-41878) Add JIRAs or messages for skipped messages

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41878: Assignee: Apache Spark (was: Hyukjin Kwon) > Add JIRAs or messages for skipped messages

[jira] [Commented] (SPARK-41878) Add JIRAs or messages for skipped messages

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654298#comment-17654298 ] Apache Spark commented on SPARK-41878: -- User 'techaddict' has created a pull request for this

[jira] [Updated] (SPARK-41878) Add JIRAs or messages for skipped messages

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41878: -- Description: Add JIRAs or Messages for all the skipped messages. (was: 5 tests pass now.

[jira] [Created] (SPARK-41878) Add JIRAs or messages for skipped messages

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41878: - Summary: Add JIRAs or messages for skipped messages Key: SPARK-41878 URL: https://issues.apache.org/jira/browse/SPARK-41878 Project: Spark Issue Type:

[jira] [Updated] (SPARK-41877) SparkSession.createDataFrame error parity

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41877: -- Description: {code:java} df = self.spark.createDataFrame( [ (1, 10, 1.0, "one"),

[jira] [Created] (SPARK-41877) SparkSession.createDataFrame error parity

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41877: - Summary: SparkSession.createDataFrame error parity Key: SPARK-41877 URL: https://issues.apache.org/jira/browse/SPARK-41877 Project: Spark Issue Type:

[jira] [Commented] (SPARK-41554) Decimal.changePrecision produces ArrayIndexOutOfBoundsException

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654294#comment-17654294 ] Apache Spark commented on SPARK-41554: -- User 'fe2s' has created a pull request for this issue:

[jira] [Created] (SPARK-41876) Implement DataFrame `toLocalIterator`

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41876: - Summary: Implement DataFrame `toLocalIterator` Key: SPARK-41876 URL: https://issues.apache.org/jira/browse/SPARK-41876 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-41859) CreateHiveTableAsSelectCommand should set the overwrite flag correctly

2023-01-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-41859: --- Assignee: Wenchen Fan > CreateHiveTableAsSelectCommand should set the overwrite flag

[jira] [Resolved] (SPARK-41859) CreateHiveTableAsSelectCommand should set the overwrite flag correctly

2023-01-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-41859. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39365

[jira] [Commented] (SPARK-36939) Add orphan migration page into list in PySpark documentation

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654292#comment-17654292 ] Dongjoon Hyun commented on SPARK-36939: --- Due to

[jira] [Updated] (SPARK-36939) Add orphan migration page into list in PySpark documentation

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36939: -- Affects Version/s: 3.3.0 (was: 3.2.0) > Add orphan migration page

[jira] [Commented] (SPARK-41862) Fix a correctness bug in existence DEFAULT value lookups for the Orc data source

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654286#comment-17654286 ] Apache Spark commented on SPARK-41862: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-41862) Fix a correctness bug in existence DEFAULT value lookups for the Orc data source

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654287#comment-17654287 ] Apache Spark commented on SPARK-41862: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-41828) Implement creating empty Dataframe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41828: Assignee: Apache Spark > Implement creating empty Dataframe >

[jira] [Commented] (SPARK-41828) Implement creating empty Dataframe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654285#comment-17654285 ] Apache Spark commented on SPARK-41828: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-41828) Implement creating empty Dataframe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41828: Assignee: (was: Apache Spark) > Implement creating empty Dataframe >

[jira] [Commented] (SPARK-41828) Implement creating empty Dataframe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654284#comment-17654284 ] Apache Spark commented on SPARK-41828: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-41850) Fix `isnan` function

2023-01-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-41850: - Assignee: Ruifeng Zheng > Fix `isnan` function > > >

[jira] [Resolved] (SPARK-41850) Fix `isnan` function

2023-01-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-41850. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39376

[jira] [Updated] (SPARK-41875) Throw proper errors in Dataset.to()

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41875: -- Description: {code:java} schema = StructType( [StructField("i", StringType(), True),

[jira] [Created] (SPARK-41875) Throw proper errors in Dataset.to()

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41875: - Summary: Throw proper errors in Dataset.to() Key: SPARK-41875 URL: https://issues.apache.org/jira/browse/SPARK-41875 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-41874) Implement DataFrame `sameSemantics`

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41874: - Summary: Implement DataFrame `sameSemantics` Key: SPARK-41874 URL: https://issues.apache.org/jira/browse/SPARK-41874 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-41872) Fix DataFrame createDataframe handling of None

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41872: -- Summary: Fix DataFrame createDataframe handling of None (was: Fix DataFrame fillna with

[jira] [Updated] (SPARK-41872) Fix DataFrame createDataframe handling of None

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41872: -- Description: {code:java} row = self.spark.createDataFrame([("Alice", None, None, None)],

[jira] [Commented] (SPARK-41821) Fix DataFrame.describe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654273#comment-17654273 ] Apache Spark commented on SPARK-41821: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-41821) Fix DataFrame.describe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41821: Assignee: (was: Apache Spark) > Fix DataFrame.describe > -- > >

[jira] [Commented] (SPARK-41821) Fix DataFrame.describe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654272#comment-17654272 ] Apache Spark commented on SPARK-41821: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-41821) Fix DataFrame.describe

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41821: Assignee: Apache Spark > Fix DataFrame.describe > -- > >

[jira] [Updated] (SPARK-41873) Implement DataFrame `pandas_api`

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41873: -- Summary: Implement DataFrame `pandas_api` (was: Implement DataFrameReader `pandas_api`) >

[jira] [Created] (SPARK-41873) Implement DataFrameReader `pandas_api`

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41873: - Summary: Implement DataFrameReader `pandas_api` Key: SPARK-41873 URL: https://issues.apache.org/jira/browse/SPARK-41873 Project: Spark Issue Type:

[jira] [Updated] (SPARK-41873) Implement DataFrame `pandas_api`

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41873: -- Description: (was: {code:java} File

[jira] [Assigned] (SPARK-41867) Selective predicate should respect InMemoryRelation

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41867: Assignee: Apache Spark > Selective predicate should respect InMemoryRelation >

[jira] [Commented] (SPARK-41867) Selective predicate should respect InMemoryRelation

2023-01-03 Thread Apache Spark (Jira)

[jira] [Assigned] (SPARK-41867) Selective predicate should respect InMemoryRelation

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41867: Assignee: (was: Apache Spark) > Selective predicate should respect InMemoryRelation

[jira] [Updated] (SPARK-41864) Fix mypy linter errors

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41864: -- Fix Version/s: 3.3.2 > Fix mypy linter errors > -- > >

[jira] [Updated] (SPARK-36883) Upgrade R version to 4.1.1 in CI images

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36883: -- Fix Version/s: 3.2.4 > Upgrade R version to 4.1.1 in CI images >

[jira] [Updated] (SPARK-41872) Fix DataFrame fillna with bool

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41872: -- Description: {code:java} row = self.spark.createDataFrame([("Alice", None, None, None)],

[jira] [Created] (SPARK-41872) Fix DataFrame fillna with bool

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41872: - Summary: Fix DataFrame fillna with bool Key: SPARK-41872 URL: https://issues.apache.org/jira/browse/SPARK-41872 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-41871) DataFrame hint parameter can be str, list, float or int

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41871: - Summary: DataFrame hint parameter can be str, list, float or int Key: SPARK-41871 URL: https://issues.apache.org/jira/browse/SPARK-41871 Project: Spark

[jira] [Updated] (SPARK-41871) DataFrame hint parameter can be str, list, float or int

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41871: -- Description: {code:java} df = self.spark.range(10e10).toDF("id") such_a_nice_list =

[jira] [Created] (SPARK-41870) Handle duplicate columns in `createDataFrame`

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41870: - Summary: Handle duplicate columns in `createDataFrame` Key: SPARK-41870 URL: https://issues.apache.org/jira/browse/SPARK-41870 Project: Spark Issue Type:

[jira] [Updated] (SPARK-41870) Handle duplicate columns in `createDataFrame`

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41870: -- Description: {code:java} df = self.spark.createDataFrame([(1, 2)], ["c", "c"]){code} Error:

[jira] [Updated] (SPARK-41867) Selective predicate should respect InMemoryRelation

2023-01-03 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-41867: -- Description: DPP and Runtime Filter require the build side has a selective predicate. It should also

[jira] [Updated] (SPARK-41869) DataFrame dropDuplicates should throw error on non list argument

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41869: -- Description: {code:java} df = self.spark.createDataFrame([("Alice", 50), ("Alice", 60)],

[jira] [Created] (SPARK-41869) DataFrame dropDuplicates should throw error on non list argument

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41869: - Summary: DataFrame dropDuplicates should throw error on non list argument Key: SPARK-41869 URL: https://issues.apache.org/jira/browse/SPARK-41869 Project: Spark

[jira] [Commented] (SPARK-41855) `createDataFrame` doesn't handle None/NaN properly

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654255#comment-17654255 ] Sandeep Singh commented on SPARK-41855: --- [~podongfeng] there is another failure which might be

[jira] [Updated] (SPARK-41856) Enable test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41856: -- Summary: Enable test_freqItems, test_input_files, test_toDF_with_schema_string,

[jira] [Commented] (SPARK-41828) Implement creating empty Dataframe

2023-01-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654254#comment-17654254 ] Ruifeng Zheng commented on SPARK-41828: --- I will take a look > Implement creating empty Dataframe

[jira] [Updated] (SPARK-41868) Support data type Duration(NANOSECOND)

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41868: -- Description: {code:java} import pandas as pd from datetime import timedelta df =

[jira] [Created] (SPARK-41867) Selective predicate should respect InMemoryRelation

2023-01-03 Thread XiDuo You (Jira)
XiDuo You created SPARK-41867: - Summary: Selective predicate should respect InMemoryRelation Key: SPARK-41867 URL: https://issues.apache.org/jira/browse/SPARK-41867 Project: Spark Issue Type:

[jira] [Created] (SPARK-41868) Support data type Duration(NANOSECOND)

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41868: - Summary: Support data type Duration(NANOSECOND) Key: SPARK-41868 URL: https://issues.apache.org/jira/browse/SPARK-41868 Project: Spark Issue Type:

[jira] [Updated] (SPARK-41866) Make `createDataFrame` support array

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41866: -- Description: {code:java} import array data = [Row(longarray=array.array("l",

[jira] [Created] (SPARK-41866) Make `createDataFrame` support array

2023-01-03 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41866: - Summary: Make `createDataFrame` support array Key: SPARK-41866 URL: https://issues.apache.org/jira/browse/SPARK-41866 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-41856) Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-03 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654239#comment-17654239 ] Sandeep Singh commented on SPARK-41856: --- [~gurwls223] for some reason its still assigned to you 

[jira] [Assigned] (SPARK-41850) Fix `isnan` function

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41850: Assignee: (was: Apache Spark) > Fix `isnan` function > > >

[jira] [Assigned] (SPARK-41850) Fix `isnan` function

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41850: Assignee: Apache Spark > Fix `isnan` function > > >

[jira] [Commented] (SPARK-41850) Fix `isnan` function

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654236#comment-17654236 ] Apache Spark commented on SPARK-41850: -- User 'zhengruifeng' has created a pull request for this

[jira] [Commented] (SPARK-41850) Fix `isnan` function

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654235#comment-17654235 ] Apache Spark commented on SPARK-41850: -- User 'zhengruifeng' has created a pull request for this

[jira] [Updated] (SPARK-41719) Spark SSLOptions sub settings should be set only when ssl is enabled

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41719: -- Affects Version/s: 3.4.0 (was: 3.2.4) > Spark SSLOptions sub

[jira] [Resolved] (SPARK-41719) Spark SSLOptions sub settings should be set only when ssl is enabled

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41719. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39221

[jira] [Assigned] (SPARK-41719) Spark SSLOptions sub settings should be set only when ssl is enabled

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41719: - Assignee: Shrikant Prasad > Spark SSLOptions sub settings should be set only when ssl

[jira] [Updated] (SPARK-41719) Spark SSLOptions sub settings should be set only when ssl is enabled

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41719: -- Priority: Minor (was: Major) > Spark SSLOptions sub settings should be set only when ssl is

[jira] [Commented] (SPARK-41833) DataFrame.collect() output parity with pyspark

2023-01-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654233#comment-17654233 ] Ruifeng Zheng commented on SPARK-41833: --- I will take a look at this one > DataFrame.collect()

[jira] [Commented] (SPARK-41772) Enable pyspark.sql.connect.column.Column.withField doctest

2023-01-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654230#comment-17654230 ] Ruifeng Zheng commented on SPARK-41772: --- this one seems related to column reference > Enable

[jira] [Commented] (SPARK-41772) Enable pyspark.sql.connect.column.Column.withField doctest

2023-01-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654229#comment-17654229 ] Ruifeng Zheng commented on SPARK-41772: --- {code:java} File

[jira] [Updated] (SPARK-41030) Upgrade Apache Ivy to 2.5.1

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41030: -- Fix Version/s: 3.2.4 > Upgrade Apache Ivy to 2.5.1 > --- > >

[jira] [Assigned] (SPARK-41865) Use pycodestyle to 2.7.0 to fix pycodestyle errors

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41865: - Assignee: Dongjoon Hyun > Use pycodestyle to 2.7.0 to fix pycodestyle errors >

[jira] [Resolved] (SPARK-41865) Use pycodestyle to 2.7.0 to fix pycodestyle errors

2023-01-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41865. --- Fix Version/s: 3.2.4 Resolution: Fixed Issue resolved by pull request 39374

[jira] [Commented] (SPARK-36124) Support set operators to be on correlation paths

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654227#comment-17654227 ] Apache Spark commented on SPARK-36124: -- User 'jchen5' has created a pull request for this issue:

[jira] [Commented] (SPARK-41821) Fix DataFrame.describe

2023-01-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654226#comment-17654226 ] Ruifeng Zheng commented on SPARK-41821: --- [~beliefer] Jia an, would you mind taking a look at this

[jira] [Assigned] (SPARK-41865) Use pycodestyle to 2.7.0 to fix pycodestyle errors

2023-01-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41865: Assignee: Apache Spark > Use pycodestyle to 2.7.0 to fix pycodestyle errors >

  1   2   3   >