[jira] [Updated] (SPARK-47891) Improve docstring of mapInPandas

2024-04-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-47891: - Description: Improve docstring of mapInPandas * "using a Python native function that takes and

[jira] [Resolved] (SPARK-47876) Improve docstring of mapInArrow

2024-04-16 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-47876. -- Resolution: Done Resolved by https://github.com/apache/spark/pull/46088 > Improve docstring

[jira] [Created] (SPARK-47876) Improve docstring of mapInArrow

2024-04-16 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-47876: Summary: Improve docstring of mapInArrow Key: SPARK-47876 URL: https://issues.apache.org/jira/browse/SPARK-47876 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-47823) Improve appName and getOrCreate usage for Spark Connect

2024-04-11 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-47823: - Description:   In Spark Connect {code:java} spark =

[jira] [Created] (SPARK-47823) Improve appName and getOrCreate usage for Spark Connect

2024-04-11 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-47823: Summary: Improve appName and getOrCreate usage for Spark Connect Key: SPARK-47823 URL: https://issues.apache.org/jira/browse/SPARK-47823 Project: Spark

[jira] [Updated] (SPARK-47677) Pandas circular import error in Python 3.10

2024-04-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-47677: - Description: {{AttributeError: partially initialized module 'pandas' has no attribute

[jira] [Created] (SPARK-47677) Pandas circular import error in Python 3.10

2024-04-01 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-47677: Summary: Pandas circular import error in Python 3.10 Key: SPARK-47677 URL: https://issues.apache.org/jira/browse/SPARK-47677 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-47276) Introduce `spark.profile.clear` for SparkSession-based profiling

2024-03-07 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-47276. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45378

[jira] [Created] (SPARK-47276) Introduce `spark.profile.clear` for SparkSession-based profiling

2024-03-04 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-47276: Summary: Introduce `spark.profile.clear` for SparkSession-based profiling Key: SPARK-47276 URL: https://issues.apache.org/jira/browse/SPARK-47276 Project: Spark

[jira] [Resolved] (SPARK-46975) Support dedicated fallback methods

2024-02-23 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-46975. -- Resolution: Done Resolved by https://github.com/apache/spark/pull/45026 > Support dedicated

[jira] [Assigned] (SPARK-46975) Support dedicated fallback methods

2024-02-23 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-46975: Assignee: Ruifeng Zheng > Support dedicated fallback methods >

[jira] [Comment Edited] (SPARK-47132) Mistake in Docstring for Pyspark's Dataframe.head()

2024-02-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819779#comment-17819779 ] Xinrong Meng edited comment on SPARK-47132 at 2/22/24 7:21 PM: ---

[jira] [Commented] (SPARK-47132) Mistake in Docstring for Pyspark's Dataframe.head()

2024-02-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819779#comment-17819779 ] Xinrong Meng commented on SPARK-47132: -- [~wunderalbert] would you double check if you set up your

[jira] [Commented] (SPARK-47132) Mistake in Docstring for Pyspark's Dataframe.head()

2024-02-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819780#comment-17819780 ] Xinrong Meng commented on SPARK-47132: -- Resolved by https://github.com/apache/spark/pull/45197. >

[jira] [Updated] (SPARK-47132) Mistake in Docstring for Pyspark's Dataframe.head()

2024-02-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-47132: - Attachment: image-2024-02-22-11-18-02-429.png > Mistake in Docstring for Pyspark's

[jira] [Updated] (SPARK-47132) Mistake in Docstring for Pyspark's Dataframe.head()

2024-02-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-47132: - Issue Type: Documentation (was: Bug) > Mistake in Docstring for Pyspark's Dataframe.head() >

[jira] [Updated] (SPARK-47132) Mistake in Docstring for Pyspark's Dataframe.head()

2024-02-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-47132: - Affects Version/s: 4.0.0 (was: 3.5.0) > Mistake in Docstring for

[jira] [Commented] (SPARK-47132) Mistake in Docstring for Pyspark's Dataframe.head()

2024-02-22 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17819777#comment-17819777 ] Xinrong Meng commented on SPARK-47132: -- I modified the ticket to Documentation (from Bug) and 4.0.0

[jira] [Created] (SPARK-47078) Documentation for SparkSession-based Profilers

2024-02-16 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-47078: Summary: Documentation for SparkSession-based Profilers Key: SPARK-47078 URL: https://issues.apache.org/jira/browse/SPARK-47078 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-47014) Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession

2024-02-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-47014: Assignee: Xinrong Meng > Implement methods dumpPerfProfiles and dumpMemoryProfiles of

[jira] [Resolved] (SPARK-47014) Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession

2024-02-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-47014. -- Resolution: Done Resolved by https://github.com/apache/spark/pull/45073 > Implement methods

[jira] [Created] (SPARK-47014) Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession

2024-02-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-47014: Summary: Implement methods dumpPerfProfiles and dumpMemoryProfiles of SparkSession Key: SPARK-47014 URL: https://issues.apache.org/jira/browse/SPARK-47014 Project:

[jira] [Assigned] (SPARK-46690) Support profiling on FlatMapCoGroupsInBatchExec

2024-02-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-46690: Assignee: Xinrong Meng > Support profiling on FlatMapCoGroupsInBatchExec >

[jira] [Resolved] (SPARK-46690) Support profiling on FlatMapCoGroupsInBatchExec

2024-02-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-46690. -- Resolution: Done Resolved by https://github.com/apache/spark/pull/45050 > Support profiling

[jira] [Resolved] (SPARK-46689) Support profiling on FlatMapGroupsInBatchExec

2024-02-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-46689. -- Resolution: Done Resolved by https://github.com/apache/spark/pull/45050 > Support profiling

[jira] [Assigned] (SPARK-46689) Support profiling on FlatMapGroupsInBatchExec

2024-02-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-46689: Assignee: Xinrong Meng > Support profiling on FlatMapGroupsInBatchExec >

[jira] [Created] (SPARK-46925) Add a warning that instructs to install memory_profiler for memory profiling

2024-01-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46925: Summary: Add a warning that instructs to install memory_profiler for memory profiling Key: SPARK-46925 URL: https://issues.apache.org/jira/browse/SPARK-46925

[jira] [Created] (SPARK-46880) Improve and test warning for Arrow-optimized Python UDF

2024-01-26 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46880: Summary: Improve and test warning for Arrow-optimized Python UDF Key: SPARK-46880 URL: https://issues.apache.org/jira/browse/SPARK-46880 Project: Spark

[jira] [Resolved] (SPARK-46467) Improve and test exceptions of TimedeltaIndex

2024-01-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-46467. -- Assignee: Xinrong Meng Resolution: Not A Problem We don't have a plan to migrate Pandas

[jira] [Created] (SPARK-46781) Test data source (pyspark.sql.datasource)

2024-01-19 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46781: Summary: Test data source (pyspark.sql.datasource) Key: SPARK-46781 URL: https://issues.apache.org/jira/browse/SPARK-46781 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46781) Test custom data source and input partition (pyspark.sql.datasource)

2024-01-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46781: - Summary: Test custom data source and input partition (pyspark.sql.datasource) (was: Test data

[jira] [Updated] (SPARK-42862) Review and fix issues in Core API docs

2024-01-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42862: - Parent: SPARK-42523 (was: SPARK-42693) > Review and fix issues in Core API docs >

[jira] [Updated] (SPARK-42863) Review and fix issues in PySpark API docs

2024-01-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42863: - Parent: SPARK-42523 (was: SPARK-42693) > Review and fix issues in PySpark API docs >

[jira] [Updated] (SPARK-42864) Review and fix issues in MLlib API docs

2024-01-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42864: - Parent: SPARK-42523 (was: SPARK-42693) > Review and fix issues in MLlib API docs >

[jira] [Updated] (SPARK-42861) Review and fix issues in SQL API docs

2024-01-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42861: - Parent: SPARK-42523 (was: SPARK-42693) > Review and fix issues in SQL API docs >

[jira] [Updated] (SPARK-42866) Review and fix issues in Spark Connect - Scala API docs

2024-01-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42866: - Parent: SPARK-42523 (was: SPARK-42693) > Review and fix issues in Spark Connect - Scala API

[jira] [Updated] (SPARK-42693) API Auditing

2024-01-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42693: - Parent: SPARK-42523 Issue Type: Sub-task (was: Story) > API Auditing > >

[jira] [Resolved] (SPARK-42523) Apache Spark 3.4 release

2024-01-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42523. -- Resolution: Done > Apache Spark 3.4 release > > >

[jira] [Created] (SPARK-46467) Improve and test exceptions of TimedeltaIndex

2023-12-20 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46467: Summary: Improve and test exceptions of TimedeltaIndex Key: SPARK-46467 URL: https://issues.apache.org/jira/browse/SPARK-46467 Project: Spark Issue Type:

[jira] [Created] (SPARK-46459) Fix bundler to 2.4.22 to unclock CI

2023-12-19 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46459: Summary: Fix bundler to 2.4.22 to unclock CI Key: SPARK-46459 URL: https://issues.apache.org/jira/browse/SPARK-46459 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-46386) Improve assertions of observation (pyspark.sql.observation)

2023-12-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46386: - Summary: Improve assertions of observation (pyspark.sql.observation) (was: Improve and test

[jira] [Updated] (SPARK-46386) Improve and test assertions of observation (pyspark.sql.observation)

2023-12-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46386: - Parent: (was: SPARK-46041) Issue Type: Improvement (was: Sub-task) > Improve and

[jira] [Updated] (SPARK-46413) Validate returnType of Arrow Python UDF

2023-12-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46413: - Description: Validate returnType of Arrow Python UDF (was: Check returnType of Arrow Python

[jira] [Updated] (SPARK-46413) Validate returnType of Arrow Python UDF

2023-12-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46413: - Summary: Validate returnType of Arrow Python UDF (was: Check returnType of Arrow Python UDF)

[jira] [Created] (SPARK-46413) Check returnType of Arrow Python UDF

2023-12-14 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46413: Summary: Check returnType of Arrow Python UDF Key: SPARK-46413 URL: https://issues.apache.org/jira/browse/SPARK-46413 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-46398) Test rangeBetween window function (pyspark.sql.window)

2023-12-13 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46398: Summary: Test rangeBetween window function (pyspark.sql.window) Key: SPARK-46398 URL: https://issues.apache.org/jira/browse/SPARK-46398 Project: Spark Issue

[jira] [Created] (SPARK-46386) Improve and test assertions of observation (pyspark.sql.observation)

2023-12-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46386: Summary: Improve and test assertions of observation (pyspark.sql.observation) Key: SPARK-46386 URL: https://issues.apache.org/jira/browse/SPARK-46386 Project: Spark

[jira] [Created] (SPARK-46385) Test aggregate functions for groups (pyspark.sql.group)

2023-12-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46385: Summary: Test aggregate functions for groups (pyspark.sql.group) Key: SPARK-46385 URL: https://issues.apache.org/jira/browse/SPARK-46385 Project: Spark

[jira] [Resolved] (SPARK-46277) Validate startup urls with the config being set

2023-12-07 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-46277. -- Resolution: Fixed Resolved by https://github.com/apache/spark/pull/44194 > Validate startup

[jira] [Assigned] (SPARK-46277) Validate startup urls with the config being set

2023-12-07 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-46277: Assignee: Xinrong Meng > Validate startup urls with the config being set >

[jira] [Updated] (SPARK-46291) Koalas Testing Migration

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46291: - Description: Test migration from Koalas to Spark repository, including setting up the testing

[jira] [Updated] (SPARK-46291) Koalas Testing Migration

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46291: - Summary: Koalas Testing Migration (was: Testing migration) > Koalas Testing Migration >

[jira] [Assigned] (SPARK-46291) Testing migration

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-46291: Assignee: Xinrong Meng > Testing migration > - > > Key:

[jira] [Resolved] (SPARK-46291) Testing migration

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-46291. -- Resolution: Done > Testing migration > - > > Key: SPARK-46291

[jira] [Updated] (SPARK-34999) Consolidate PySpark testing utils

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-34999: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Consolidate PySpark testing utils >

[jira] [Updated] (SPARK-35012) Port Koalas DataFrame related unit tests into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35012: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port Koalas DataFrame related unit tests into

[jira] [Updated] (SPARK-35300) Standardize module name in install.rst

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35300: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Standardize module name in install.rst >

[jira] [Updated] (SPARK-35034) Port Koalas miscellaneous unit tests into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35034: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port Koalas miscellaneous unit tests into

[jira] [Updated] (SPARK-35035) Port Koalas internal implementation unit tests into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35035: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port Koalas internal implementation unit tests

[jira] [Updated] (SPARK-35040) Remove Spark-version related codes from test codes.

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35040: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Remove Spark-version related codes from test

[jira] [Updated] (SPARK-35098) Revisit pandas-on-Spark test cases that are disabled because of pandas nondeterministic return values

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35098: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Revisit pandas-on-Spark test cases that are

[jira] [Updated] (SPARK-35033) Port Koalas plot unit tests into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35033: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port Koalas plot unit tests into PySpark >

[jira] [Updated] (SPARK-35032) Port Koalas Index unit tests into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35032: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port Koalas Index unit tests into PySpark >

[jira] [Updated] (SPARK-35031) Port Koalas operations on different frames tests into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-35031: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port Koalas operations on different frames

[jira] [Updated] (SPARK-34996) Port Koalas Series related unit tests into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-34996: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port Koalas Series related unit tests into

[jira] [Updated] (SPARK-34887) Port/integrate Koalas dependencies into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-34887: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port/integrate Koalas dependencies into PySpark

[jira] [Updated] (SPARK-34886) Port/integrate Koalas DataFrame unit test into PySpark

2023-12-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-34886: - Parent Issue: SPARK-46291 (was: SPARK-34849) > Port/integrate Koalas DataFrame unit test into

[jira] [Created] (SPARK-46291) Testing migration

2023-12-06 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46291: Summary: Testing migration Key: SPARK-46291 URL: https://issues.apache.org/jira/browse/SPARK-46291 Project: Spark Issue Type: Umbrella Components:

[jira] [Updated] (SPARK-46277) Validate startup urls with the config to set

2023-12-05 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46277: - Attachment: image-2023-12-05-15-39-08-830.png > Validate startup urls with the config to set >

[jira] [Updated] (SPARK-46277) Validate startup urls with the config to set

2023-12-05 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46277: - Description: !image-2023-12-05-15-39-08-830.png! > Validate startup urls with the config to set

[jira] [Updated] (SPARK-46277) Validate startup urls with the config being set

2023-12-05 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-46277: - Summary: Validate startup urls with the config being set (was: Validate startup urls with the

[jira] [Created] (SPARK-46277) Validate startup urls with the config to set

2023-12-05 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46277: Summary: Validate startup urls with the config to set Key: SPARK-46277 URL: https://issues.apache.org/jira/browse/SPARK-46277 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-46252) Improve test coverage of memory_profiler.py

2023-12-04 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46252: Summary: Improve test coverage of memory_profiler.py Key: SPARK-46252 URL: https://issues.apache.org/jira/browse/SPARK-46252 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-44560) Improve tests and documentation for Arrow Python UDF

2023-07-27 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-44560. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44560) Improve tests and documentation for Arrow Python UDF

2023-07-27 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-44560: Assignee: Xinrong Meng > Improve tests and documentation for Arrow Python UDF >

[jira] [Created] (SPARK-44560) Improve tests and documentation for Arrow Python UDF

2023-07-26 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-44560: Summary: Improve tests and documentation for Arrow Python UDF Key: SPARK-44560 URL: https://issues.apache.org/jira/browse/SPARK-44560 Project: Spark Issue

[jira] [Updated] (SPARK-44486) Implement PyArrow `self_destruct` feature for `toPandas`

2023-07-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-44486: - Description: Implement PyArrow `self_destruct` feature for `toPandas` To make the Spark

[jira] [Updated] (SPARK-44486) Implement PyArrow `self_destruct` feature for `toPandas`

2023-07-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-44486: - Description: Implement PyArrow `self_destruct` feature for `toPandas`   Now the Spark

[jira] [Created] (SPARK-44486) Implement PyArrow `self_destruct` feature for `toPandas`

2023-07-19 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-44486: Summary: Implement PyArrow `self_destruct` feature for `toPandas` Key: SPARK-44486 URL: https://issues.apache.org/jira/browse/SPARK-44486 Project: Spark

[jira] [Assigned] (SPARK-44446) Add checks for expected list type special cases

2023-07-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-6: Assignee: Amanda Liu > Add checks for expected list type special cases >

[jira] [Resolved] (SPARK-44446) Add checks for expected list type special cases

2023-07-17 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-6?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-6. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42023

[jira] [Commented] (SPARK-44264) DeepSpeed Distrobutor

2023-07-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743293#comment-17743293 ] Xinrong Meng commented on SPARK-44264: -- Issue resolved by pull request

[jira] [Resolved] (SPARK-44398) Scala foreachBatch API in Streaming Spark Connect

2023-07-13 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-44398. -- Resolution: Fixed Issue resolved by pull request 41969

[jira] [Assigned] (SPARK-44398) Scala foreachBatch API in Streaming Spark Connect

2023-07-13 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-44398: Assignee: Raghu Angadi > Scala foreachBatch API in Streaming Spark Connect >

[jira] [Updated] (SPARK-44401) Arrow Python UDF Use Guide

2023-07-12 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-44401: - Component/s: Documentation > Arrow Python UDF Use Guide > -- > >

[jira] [Created] (SPARK-44401) Arrow Python UDF Use Guide

2023-07-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-44401: Summary: Arrow Python UDF Use Guide Key: SPARK-44401 URL: https://issues.apache.org/jira/browse/SPARK-44401 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-44399) Import SparkSession in Python UDF only when useArrow is None

2023-07-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-44399: Summary: Import SparkSession in Python UDF only when useArrow is None Key: SPARK-44399 URL: https://issues.apache.org/jira/browse/SPARK-44399 Project: Spark

[jira] [Assigned] (SPARK-44150) Explicit Arrow casting for mismatched return type in Arrow Python UDF

2023-06-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-44150: Assignee: Xinrong Meng > Explicit Arrow casting for mismatched return type in Arrow

[jira] [Resolved] (SPARK-44150) Explicit Arrow casting for mismatched return type in Arrow Python UDF

2023-06-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-44150. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41503

[jira] [Created] (SPARK-44150) Explicit Arrow casting for mismatched return type in Arrow Python UDF

2023-06-22 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-44150: Summary: Explicit Arrow casting for mismatched return type in Arrow Python UDF Key: SPARK-44150 URL: https://issues.apache.org/jira/browse/SPARK-44150 Project: Spark

[jira] [Updated] (SPARK-40307) Introduce Arrow Python UDFs

2023-06-16 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Affects Version/s: (was: 3.4.0) > Introduce Arrow Python UDFs > ---

[jira] [Updated] (SPARK-43440) Support registration of an Arrow Python UDF

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43440: - Summary: Support registration of an Arrow Python UDF (was: Support registration of an

[jira] [Updated] (SPARK-43893) Non-atomic data type support in Arrow Python UDF

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43893: - Summary: Non-atomic data type support in Arrow Python UDF (was: Non-atomic data type support

[jira] [Updated] (SPARK-43412) Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow Python UDFs

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43412: - Summary: Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow Python UDFs (was: Introduce

[jira] [Updated] (SPARK-43082) Arrow Python UDFs in Spark Connect

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43082: - Summary: Arrow Python UDFs in Spark Connect (was: Arrow-optimized Python UDFs in Spark

[jira] [Updated] (SPARK-42893) Block Arrow Python UDFs

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42893: - Summary: Block Arrow Python UDFs (was: Block Arrow-optimized Python UDFs) > Block Arrow Python

[jira] [Updated] (SPARK-40307) Introduce Arrow Python UDFs

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Summary: Introduce Arrow Python UDFs (was: Introduce Arrow-optimized Python UDFs) > Introduce

[jira] [Updated] (SPARK-43903) Improve ArrayType input support in Arrow Python UDF

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43903: - Summary: Improve ArrayType input support in Arrow Python UDF (was: Improve ArrayType input

[jira] [Updated] (SPARK-43903) Improve ArrayType input support in Arrow-optimized Python UDF

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43903: - Summary: Improve ArrayType input support in Arrow-optimized Python UDF (was: Non-atomic data

[jira] [Updated] (SPARK-43893) Non-atomic data type support in Arrow-optimized Python UDF

2023-06-14 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43893: - Summary: Non-atomic data type support in Arrow-optimized Python UDF (was: StructType

  1   2   3   4   5   6   7   8   9   10   >