[jira] [Assigned] (SPARK-43893) StructType input/output support in Arrow-optimized Python UDF

2023-06-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-43893: Assignee: Xinrong Meng > StructType input/output support in Arrow-optimized Python UDF >

[jira] [Resolved] (SPARK-43893) StructType input/output support in Arrow-optimized Python UDF

2023-06-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-43893. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41321

[jira] [Updated] (SPARK-43903) Non-atomic data type support in Arrow-optimized Python UDF

2023-05-31 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43903: - Summary: Non-atomic data type support in Arrow-optimized Python UDF (was: Standardize

[jira] [Created] (SPARK-43903) Standardize ArrayType conversion for Python UDF

2023-05-31 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43903: Summary: Standardize ArrayType conversion for Python UDF Key: SPARK-43903 URL: https://issues.apache.org/jira/browse/SPARK-43903 Project: Spark Issue Type:

[jira] [Created] (SPARK-43893) StructType input/output support in Arrow-optimized Python UDF

2023-05-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43893: Summary: StructType input/output support in Arrow-optimized Python UDF Key: SPARK-43893 URL: https://issues.apache.org/jira/browse/SPARK-43893 Project: Spark

[jira] [Created] (SPARK-43886) Accept generics tuple as typing hints in Pandas UDF

2023-05-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43886: Summary: Accept generics tuple as typing hints in Pandas UDF Key: SPARK-43886 URL: https://issues.apache.org/jira/browse/SPARK-43886 Project: Spark Issue

[jira] [Created] (SPARK-43804) Test on nested structs support in Pandas UDF

2023-05-25 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43804: Summary: Test on nested structs support in Pandas UDF Key: SPARK-43804 URL: https://issues.apache.org/jira/browse/SPARK-43804 Project: Spark Issue Type:

[jira] [Updated] (SPARK-43545) Support Nested Timestamp Types

2023-05-25 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43545: - Summary: Support Nested Timestamp Types (was: Remove outdated

[jira] (SPARK-43543) Standardize Nested Complex DataTypes Support

2023-05-24 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43543 ] Xinrong Meng deleted comment on SPARK-43543: -- was (Author: xinrongm): Issue resolved by pull request 41147 [https://github.com/apache/spark/pull/41147] > Standardize Nested Complex

[jira] [Commented] (SPARK-43544) Fix nested MapType behavior in Pandas UDF

2023-05-24 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725949#comment-17725949 ] Xinrong Meng commented on SPARK-43544: -- Resolved by https://github.com/apache/spark/pull/41147. >

[jira] [Assigned] (SPARK-43544) Fix nested MapType behavior in Pandas UDF

2023-05-24 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-43544: Assignee: Xinrong Meng > Fix nested MapType behavior in Pandas UDF >

[jira] [Resolved] (SPARK-43544) Fix nested MapType behavior in Pandas UDF

2023-05-24 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-43544. -- Resolution: Done > Fix nested MapType behavior in Pandas UDF >

[jira] [Updated] (SPARK-43546) Complete parity tests of Pandas UDF

2023-05-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43546: - Summary: Complete parity tests of Pandas UDF (was: Complete Pandas UDF parity tests) >

[jira] [Created] (SPARK-43734) Expression "(v)" within a window function doesn't raise a AnalysisException

2023-05-22 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43734: Summary: Expression "(v)" within a window function doesn't raise a AnalysisException Key: SPARK-43734 URL: https://issues.apache.org/jira/browse/SPARK-43734 Project:

[jira] [Created] (SPARK-43727) Parity returnType check in Spark Connect

2023-05-22 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43727: Summary: Parity returnType check in Spark Connect Key: SPARK-43727 URL: https://issues.apache.org/jira/browse/SPARK-43727 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-43543) Standardize Nested Complex DataTypes Support

2023-05-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-43543. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41147

[jira] [Assigned] (SPARK-43543) Standardize Nested Complex DataTypes Support

2023-05-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-43543: Assignee: Xinrong Meng > Standardize Nested Complex DataTypes Support >

[jira] [Created] (SPARK-43579) Cache the converter between Arrow and pandas for reuse

2023-05-18 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43579: Summary: Cache the converter between Arrow and pandas for reuse Key: SPARK-43579 URL: https://issues.apache.org/jira/browse/SPARK-43579 Project: Spark Issue

[jira] [Updated] (SPARK-43544) Fix nested MapType behavior in Pandas UDF

2023-05-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43544: - Summary: Fix nested MapType behavior in Pandas UDF (was: Standardize nested non-atomic input

[jira] [Created] (SPARK-43546) Complete Pandas UDF parity tests

2023-05-17 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43546: Summary: Complete Pandas UDF parity tests Key: SPARK-43546 URL: https://issues.apache.org/jira/browse/SPARK-43546 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-43545) Remove outdated UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION

2023-05-17 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43545: Summary: Remove outdated UNSUPPORTED_DATA_TYPE_FOR_ARROW_CONVERSION Key: SPARK-43545 URL: https://issues.apache.org/jira/browse/SPARK-43545 Project: Spark

[jira] [Created] (SPARK-43544) Standardize nested non-atomic input type support in Pandas UDF

2023-05-17 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43544: Summary: Standardize nested non-atomic input type support in Pandas UDF Key: SPARK-43544 URL: https://issues.apache.org/jira/browse/SPARK-43544 Project: Spark

[jira] [Created] (SPARK-43543) Standardize Nested Complex DataTypes Support

2023-05-17 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43543: Summary: Standardize Nested Complex DataTypes Support Key: SPARK-43543 URL: https://issues.apache.org/jira/browse/SPARK-43543 Project: Spark Issue Type:

[jira] [Updated] (SPARK-43440) Support registration of an Arrow-optimized Python UDF

2023-05-10 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43440: - Description: Currently, when users register an Arrow-optimized Python UDF, it will be

[jira] [Created] (SPARK-43440) Support registration of an Arrow-optimized Python UDF

2023-05-10 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43440: Summary: Support registration of an Arrow-optimized Python UDF Key: SPARK-43440 URL: https://issues.apache.org/jira/browse/SPARK-43440 Project: Spark Issue

[jira] [Commented] (SPARK-42523) Apache Spark 3.4 release

2023-05-10 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17721549#comment-17721549 ] Xinrong Meng commented on SPARK-42523: -- I am wondering if we shall keep the ticket open for minor

[jira] [Resolved] (SPARK-43412) Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow-optimized Python UDFs

2023-05-10 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-43412. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41053

[jira] [Created] (SPARK-43412) Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow-optimized Python UDFs

2023-05-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43412: Summary: Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow-optimized Python UDFs Key: SPARK-43412 URL: https://issues.apache.org/jira/browse/SPARK-43412 Project:

[jira] [Commented] (SPARK-41971) `toPandas` should support duplicate filed names when arrow-optimization is on

2023-05-04 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719463#comment-17719463 ] Xinrong Meng commented on SPARK-41971: -- Hi [~nikj] , the issue has been resolved. Feel free to pick

[jira] [Assigned] (SPARK-41971) `toPandas` should support duplicate filed names when arrow-optimization is on

2023-05-04 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-41971: Assignee: Takuya Ueshin > `toPandas` should support duplicate filed names when

[jira] [Resolved] (SPARK-41971) `toPandas` should support duplicate filed names when arrow-optimization is on

2023-05-04 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-41971. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40988

[jira] [Resolved] (SPARK-43032) Add StreamingQueryManager API

2023-05-02 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-43032. -- Assignee: Wei Liu Resolution: Done Resolved by

[jira] [Updated] (SPARK-39892) Use ArrowType.Decimal(precision, scale, bitWidth) instead of ArrowType.Decimal(precision, scale)

2023-04-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39892: - Fix Version/s: 3.5.0 (was: 3.4.0) > Use ArrowType.Decimal(precision,

[jira] [Updated] (SPARK-41259) Spark-sql cli query results should correspond to schema

2023-04-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41259: - Fix Version/s: 3.5.0 (was: 3.4.0) > Spark-sql cli query results should

[jira] [Updated] (SPARK-39814) Use AmazonKinesisClientBuilder.withCredentials instead of new AmazonKinesisClient(credentials)

2023-04-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39814: - Fix Version/s: 3.5.0 (was: 3.4.0) > Use

[jira] [Updated] (SPARK-39136) JDBCTable support properties

2023-04-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39136: - Fix Version/s: 3.5.0 (was: 3.4.0) > JDBCTable support properties >

[jira] [Updated] (SPARK-37935) Migrate onto error classes

2023-04-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37935: - Fix Version/s: 3.5.0 (was: 3.4.0) > Migrate onto error classes >

[jira] [Updated] (SPARK-42169) Implement code generation for `to_csv` function (StructsToCsv)

2023-04-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42169: - Fix Version/s: 3.5.0 (was: 3.4.0) > Implement code generation for

[jira] [Updated] (SPARK-38945) simply KEYTAB and PRINCIPAL in KerberosConfDriverFeatureStep

2023-04-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38945: - Fix Version/s: 3.5.0 (was: 3.4.0) > simply KEYTAB and PRINCIPAL in

[jira] [Created] (SPARK-43082) Arrow-optimized Python UDFs in Spark Connect

2023-04-10 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43082: Summary: Arrow-optimized Python UDFs in Spark Connect Key: SPARK-43082 URL: https://issues.apache.org/jira/browse/SPARK-43082 Project: Spark Issue Type:

[jira] [Updated] (SPARK-39696) Uncaught exception in thread executor-heartbeater java.util.ConcurrentModificationException: mutation occurred during iteration

2023-04-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39696: - Priority: Blocker (was: Major) > Uncaught exception in thread executor-heartbeater >

[jira] [Created] (SPARK-43041) Restore constructors of exceptions for compatibility in connector API

2023-04-05 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-43041: Summary: Restore constructors of exceptions for compatibility in connector API Key: SPARK-43041 URL: https://issues.apache.org/jira/browse/SPARK-43041 Project: Spark

[jira] [Updated] (SPARK-43011) array_insert should fail with 0 index

2023-04-04 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43011: - Priority: Blocker (was: Major) > array_insert should fail with 0 index >

[jira] [Updated] (SPARK-43009) Parameterized sql() with constants

2023-04-04 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43009: - Priority: Blocker (was: Major) > Parameterized sql() with constants >

[jira] [Resolved] (SPARK-42693) API Auditing

2023-03-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42693. -- Resolution: Done > API Auditing > > > Key: SPARK-42693 >

[jira] [Resolved] (SPARK-42862) Review and fix issues in Core API docs

2023-03-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42862. -- Resolution: Resolved > Review and fix issues in Core API docs >

[jira] [Resolved] (SPARK-42866) Review and fix issues in Spark Connect - Scala API docs

2023-03-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42866. -- Resolution: Won't Do There doesn't seem to be a separate API doc for Spark Connect Scala

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Description: There are derivative APIs which depend on the implementation of Pandas UDFs:

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Attachment: image-2023-03-29-11-40-44-318.png > Support for Pandas/Arrow Functions API >

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: User-defined Functions in Python consist of (pickled) Python UDFs and

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Description: See design doc

[jira] [Resolved] (SPARK-42393) Support for Pandas/Arrow Functions API

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42393. -- Resolution: Resolved > Support for Pandas/Arrow Functions API >

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

2023-03-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Affects Version/s: (was: 3.5.0) > Support for Pandas/Arrow Functions API >

[jira] [Resolved] (SPARK-42891) Implement CoGrouped Map API

2023-03-24 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42891. -- Assignee: Xinrong Meng Resolution: Fixed Resolved by

[jira] [Updated] (SPARK-42693) API Auditing

2023-03-23 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42693: - Description: Audit user-facing API of Spark 3.4. The main goal is to ensure public API docs to

[jira] [Updated] (SPARK-42693) API Auditing

2023-03-23 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42693: - Description: Audit user-facing API of Spark 3.4. The main goal is to ensure public API docs to

[jira] [Created] (SPARK-42908) Raise RuntimeError if SparkContext is not initialized when parsing DDL-formatted type strings

2023-03-23 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-42908: Summary: Raise RuntimeError if SparkContext is not initialized when parsing DDL-formatted type strings Key: SPARK-42908 URL: https://issues.apache.org/jira/browse/SPARK-42908

[jira] [Reopened] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reopened SPARK-40307: -- > Introduce Arrow-optimized Python UDFs > - > >

[jira] [Updated] (SPARK-40307) Introduce Arrow-optimized Python UDFs

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Affects Version/s: 3.5.0 > Introduce Arrow-optimized Python UDFs >

[jira] [Updated] (SPARK-42893) Block the usage of Arrow-optimized Python UDFs

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42893: - Description: Considering the upcoming improvements on the result inconsistencies between

[jira] [Updated] (SPARK-42893) Block Arrow-optimized Python UDFs

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42893: - Summary: Block Arrow-optimized Python UDFs (was: Block the usage of Arrow-optimized Python

[jira] [Created] (SPARK-42893) Block the usage of Arrow-optimized Python UDFs

2023-03-21 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-42893: Summary: Block the usage of Arrow-optimized Python UDFs Key: SPARK-42893 URL: https://issues.apache.org/jira/browse/SPARK-42893 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42340) Implement Grouped Map API

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42340: - Summary: Implement Grouped Map API (was: Implement GroupedData.applyInPandas) > Implement

[jira] [Created] (SPARK-42891) Implement CoGrouped Map API

2023-03-21 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-42891: Summary: Implement CoGrouped Map API Key: SPARK-42891 URL: https://issues.apache.org/jira/browse/SPARK-42891 Project: Spark Issue Type: Sub-task

[jira] [Comment Edited] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17703123#comment-17703123 ] Xinrong Meng edited comment on SPARK-40327 at 3/21/23 9:48 AM: --- All

[jira] [Updated] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40327: - Affects Version/s: 3.5.0 (was: 3.4.0) > Increase pandas API coverage

[jira] [Commented] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17703123#comment-17703123 ] Xinrong Meng commented on SPARK-40327: -- Hi, all resolved issues are moved to

[jira] [Updated] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40327: - Fix Version/s: (was: 3.4.0) > Increase pandas API coverage for pandas API on Spark >

[jira] [Updated] (SPARK-40340) Implement `Expanding.sem`.

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40340: - Parent: SPARK-40327 (was: SPARK-42882) > Implement `Expanding.sem`. >

[jira] [Updated] (SPARK-40341) Implement `Rolling.median`.

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40341: - Parent: SPARK-40327 (was: SPARK-42882) > Implement `Rolling.median`. >

[jira] [Commented] (SPARK-39199) Implement pandas API missing parameters

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17703121#comment-17703121 ] Xinrong Meng commented on SPARK-39199: -- Please see

[jira] [Resolved] (SPARK-42883) Implement Pandas API Missing Parameters

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42883. -- Resolution: Resolved > Implement Pandas API Missing Parameters >

[jira] [Updated] (SPARK-38552) Implement `keep` parameter of `frame.nlargest/nsmallest` to decide how to resolve ties

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38552: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `keep` parameter of

[jira] [Resolved] (SPARK-42882) Pandas API Coverage Improvements

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42882. -- Resolution: Resolved > Pandas API Coverage Improvements > >

[jira] [Updated] (SPARK-38938) Implement `inplace` and `columns` parameters of `Series.drop`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38938: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `inplace` and `columns` parameters of

[jira] [Updated] (SPARK-38479) Add `Series.duplicated` to indicate duplicate Series values.

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38479: - Parent: SPARK-42883 (was: SPARK-39199) > Add `Series.duplicated` to indicate duplicate Series

[jira] [Updated] (SPARK-38518) Implement `skipna` of `Series.all/Index.all` to exclude NA/null values

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38518: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `skipna` of `Series.all/Index.all` to

[jira] [Updated] (SPARK-39189) interpolate supports limit_area

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39189: - Parent: SPARK-42883 (was: SPARK-39199) > interpolate supports limit_area >

[jira] [Updated] (SPARK-38903) Implement `ignore_index` of `Series.sort_values` and `Series.sort_index`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38903: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `ignore_index` of `Series.sort_values` and

[jira] [Updated] (SPARK-38943) EWM support ignore_na

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38943: - Parent: SPARK-42883 (was: SPARK-39199) > EWM support ignore_na > - > >

[jira] [Updated] (SPARK-39907) Implement axis and skipna of Series.argmin

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39907: - Parent: SPARK-42883 (was: SPARK-39199) > Implement axis and skipna of Series.argmin >

[jira] [Updated] (SPARK-38765) Implement `inplace` parameter of `Series.clip`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38765: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `inplace` parameter of `Series.clip` >

[jira] [Updated] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38686: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `keep` parameter of

[jira] [Updated] (SPARK-38704) Support string `inclusive` parameter of `Series.between`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38704: - Parent: SPARK-42883 (was: SPARK-39199) > Support string `inclusive` parameter of

[jira] [Updated] (SPARK-39201) Implement `ignore_index` of `DataFrame.explode` and `DataFrame.drop_duplicates`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-39201: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `ignore_index` of `DataFrame.explode` and

[jira] [Updated] (SPARK-38387) Support `na_action` and Series input correspondence in `Series.map`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38387: - Parent: SPARK-42883 (was: SPARK-39199) > Support `na_action` and Series input correspondence

[jira] [Updated] (SPARK-38576) Implement `numeric_only` parameter for `DataFrame/Series.rank` to rank numeric columns only

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38576: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `numeric_only` parameter for

[jira] [Updated] (SPARK-38608) Implement `bool_only` parameter of `DataFrame.all` and`DataFrame.any`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38608: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `bool_only` parameter of `DataFrame.all`

[jira] [Updated] (SPARK-38726) Support `how` parameter of `MultiIndex.dropna`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38726: - Parent: SPARK-42883 (was: SPARK-39199) > Support `how` parameter of `MultiIndex.dropna` >

[jira] [Updated] (SPARK-38441) Support string and bool `regex` in `Series.replace`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38441: - Parent: SPARK-42883 (was: SPARK-39199) > Support string and bool `regex` in `Series.replace` >

[jira] [Updated] (SPARK-38989) Implement `ignore_index` of `DataFrame/Series.sample`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38989: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `ignore_index` of `DataFrame/Series.sample`

[jira] [Updated] (SPARK-38793) Support `return_indexer` parameter of `Index/MultiIndex.sort_values`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38793: - Parent: SPARK-42883 (was: SPARK-39199) > Support `return_indexer` parameter of

[jira] [Updated] (SPARK-38491) Support `ignore_index` of `Series.sort_values`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38491: - Parent: SPARK-42883 (was: SPARK-39199) > Support `ignore_index` of `Series.sort_values` >

[jira] [Updated] (SPARK-38863) Implement `skipna` parameter of `DataFrame.all`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38863: - Parent: SPARK-42883 (was: SPARK-39199) > Implement `skipna` parameter of `DataFrame.all` >

[jira] [Updated] (SPARK-38937) interpolate support param `limit_direction`

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38937: - Parent: SPARK-42883 (was: SPARK-39199) > interpolate support param `limit_direction` >

[jira] [Updated] (SPARK-38763) Pandas API on spark Can`t apply lamda to columns.

2023-03-21 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38763: - Parent: SPARK-42883 (was: SPARK-39199) > Pandas API on spark Can`t apply lamda to columns. >

<    1   2   3   4   5   6   7   8   9   10   >