[jira] [Created] (SPARK-38765) Implement `inplace` parameter of `Series.clip`

2022-04-01 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38765: Summary: Implement `inplace` parameter of `Series.clip` Key: SPARK-38765 URL: https://issues.apache.org/jira/browse/SPARK-38765 Project: Spark Issue Type:

[jira] [Created] (SPARK-38726) Support `how` parameter of `MultiIndex.dropna`

2022-03-31 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38726: Summary: Support `how` parameter of `MultiIndex.dropna` Key: SPARK-38726 URL: https://issues.apache.org/jira/browse/SPARK-38726 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38704) Support string `inclusive` parameter of `Series.between`

2022-03-30 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38704: - Description: Support string `inclusive` parameter of `Series.between` > Support string

[jira] [Created] (SPARK-38704) Support string `inclusive` parameter of `Series.between`

2022-03-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38704: Summary: Support string `inclusive` parameter of `Series.between` Key: SPARK-38704 URL: https://issues.apache.org/jira/browse/SPARK-38704 Project: Spark

[jira] [Created] (SPARK-38686) Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`

2022-03-29 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38686: Summary: Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates` Key: SPARK-38686 URL: https://issues.apache.org/jira/browse/SPARK-38686 Project: Spark

[jira] [Updated] (SPARK-38608) [SPARK-38608][PYTHON] Implement `bool_only` parameter of `DataFrame.all` and`DataFrame.any`

2022-03-20 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38608: - Summary: [SPARK-38608][PYTHON] Implement `bool_only` parameter of `DataFrame.all`

[jira] [Updated] (SPARK-38608) [SPARK-38608][PYTHON] Implement `bool_only` parameter of `DataFrame.all` and`DataFrame.any`

2022-03-20 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38608: - Description: Implement `bool_only` parameter of `DataFrame.all` and`DataFrame.any` to include

[jira] [Updated] (SPARK-38608) [SPARK-38608][PYTHON] Implement `bool_only` parameter of `DataFrame.all` and`DataFrame.any`

2022-03-20 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38608: - Description: [SPARK-38608][PYTHON] Implement `bool_only` parameter of `DataFrame.all`

[jira] [Created] (SPARK-38608) Implement `bool_only` parameter of `DataFrame.all` to include only boolean columns

2022-03-20 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38608: Summary: Implement `bool_only` parameter of `DataFrame.all` to include only boolean columns Key: SPARK-38608 URL: https://issues.apache.org/jira/browse/SPARK-38608

[jira] [Created] (SPARK-38576) Implement `numeric_only` parameter for `DataFrame/Series.rank` to rank numeric columns only

2022-03-16 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38576: Summary: Implement `numeric_only` parameter for `DataFrame/Series.rank` to rank numeric columns only Key: SPARK-38576 URL: https://issues.apache.org/jira/browse/SPARK-38576

[jira] [Created] (SPARK-38552) Implement `keep` parameter of `frame.nlargest/nsmallest` to decide how to resolve ties

2022-03-14 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38552: Summary: Implement `keep` parameter of `frame.nlargest/nsmallest` to decide how to resolve ties Key: SPARK-38552 URL: https://issues.apache.org/jira/browse/SPARK-38552

[jira] [Created] (SPARK-38518) Implement `skipna` of `Series.all/Index.all` to exclude NA/null values

2022-03-10 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38518: Summary: Implement `skipna` of `Series.all/Index.all` to exclude NA/null values Key: SPARK-38518 URL: https://issues.apache.org/jira/browse/SPARK-38518 Project:

[jira] [Created] (SPARK-38491) Support `ignore_index` of `Series.sort_values`

2022-03-09 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38491: Summary: Support `ignore_index` of `Series.sort_values` Key: SPARK-38491 URL: https://issues.apache.org/jira/browse/SPARK-38491 Project: Spark Issue Type:

[jira] [Created] (SPARK-38487) Fix docstrings of nlargest/nsmallest of DataFrame

2022-03-09 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38487: Summary: Fix docstrings of nlargest/nsmallest of DataFrame Key: SPARK-38487 URL: https://issues.apache.org/jira/browse/SPARK-38487 Project: Spark Issue

[jira] [Updated] (SPARK-38479) Add `Series.duplicated` to indicate duplicate Series values.

2022-03-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38479: - Issue Type: New Feature (was: Improvement) > Add `Series.duplicated` to indicate duplicate

[jira] [Created] (SPARK-38479) Add `Series.duplicated` to indicate duplicate Series values.

2022-03-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38479: Summary: Add `Series.duplicated` to indicate duplicate Series values. Key: SPARK-38479 URL: https://issues.apache.org/jira/browse/SPARK-38479 Project: Spark

[jira] [Created] (SPARK-38441) Support string and bool `regex` in `Series.replace`

2022-03-07 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38441: Summary: Support string and bool `regex` in `Series.replace` Key: SPARK-38441 URL: https://issues.apache.org/jira/browse/SPARK-38441 Project: Spark Issue

[jira] [Created] (SPARK-38400) Enable Series.rename to change index labels

2022-03-02 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38400: Summary: Enable Series.rename to change index labels Key: SPARK-38400 URL: https://issues.apache.org/jira/browse/SPARK-38400 Project: Spark Issue Type:

[jira] [Created] (SPARK-38387) Support `na_action` and Series input correspondence in `Series.map`

2022-03-01 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38387: Summary: Support `na_action` and Series input correspondence in `Series.map` Key: SPARK-38387 URL: https://issues.apache.org/jira/browse/SPARK-38387 Project: Spark

[jira] [Updated] (SPARK-38345) Introduce SQL function ARRAY_SIZE

2022-02-27 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38345: - Description: Counting elements within an array is a common use case. ARRAY_SIZE ensures the

[jira] [Created] (SPARK-38345) Introduce SQL function ARRAY_SIZE

2022-02-27 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38345: Summary: Introduce SQL function ARRAY_SIZE Key: SPARK-38345 URL: https://issues.apache.org/jira/browse/SPARK-38345 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-38345) Introduce SQL function ARRAY_SIZE

2022-02-27 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17498729#comment-17498729 ] Xinrong Meng commented on SPARK-38345: -- I am working on that. > Introduce SQL function ARRAY_SIZE

[jira] [Updated] (SPARK-38225) Adjust input `format` of function `to_binary`

2022-02-16 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38225: - Description: Currently, function to_binary doesn't deal with the non-string `format` parameter

[jira] [Updated] (SPARK-38225) Adjust input `format` of function `to_binary`

2022-02-16 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-38225: - Summary: Adjust input `format` of function `to_binary` (was: Complete input validation of

[jira] [Commented] (SPARK-38225) Complete input validation of function to_binary

2022-02-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492984#comment-17492984 ] Xinrong Meng commented on SPARK-38225: -- I am working on that. > Complete input validation of

[jira] [Created] (SPARK-38225) Complete input validation of function to_binary

2022-02-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38225: Summary: Complete input validation of function to_binary Key: SPARK-38225 URL: https://issues.apache.org/jira/browse/SPARK-38225 Project: Spark Issue Type:

[jira] [Commented] (SPARK-38184) Fix malformatted ExpressionDescription of `decode`

2022-02-10 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490680#comment-17490680 ] Xinrong Meng commented on SPARK-38184: -- Working on this. > Fix malformatted ExpressionDescription

[jira] [Created] (SPARK-38184) Fix malformatted ExpressionDescription of `decode`

2022-02-10 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38184: Summary: Fix malformatted ExpressionDescription of `decode` Key: SPARK-38184 URL: https://issues.apache.org/jira/browse/SPARK-38184 Project: Spark Issue

[jira] [Updated] (SPARK-37773) Disable certain doctests of `ps.to_timedelta` for pandas<=1.0.5

2021-12-28 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37773: - Summary: Disable certain doctests of `ps.to_timedelta` for pandas<=1.0.5 (was: Disable certain

[jira] [Created] (SPARK-37773) Disable certain doctests for pandas<=1.0.5

2021-12-28 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37773: Summary: Disable certain doctests for pandas<=1.0.5 Key: SPARK-37773 URL: https://issues.apache.org/jira/browse/SPARK-37773 Project: Spark Issue Type:

[jira] [Created] (SPARK-37701) Implement `ps.to_timedelta` method

2021-12-20 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37701: Summary: Implement `ps.to_timedelta` method Key: SPARK-37701 URL: https://issues.apache.org/jira/browse/SPARK-37701 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37673) Implement `ps.timedelta_range` method

2021-12-16 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37673: Summary: Implement `ps.timedelta_range` method Key: SPARK-37673 URL: https://issues.apache.org/jira/browse/SPARK-37673 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37651) Use existing active Spark session in all places of pandas API on Spark

2021-12-14 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37651: Summary: Use existing active Spark session in all places of pandas API on Spark Key: SPARK-37651 URL: https://issues.apache.org/jira/browse/SPARK-37651 Project:

[jira] [Created] (SPARK-37563) Implement days, seconds, microseconds properties of TimedeltaIndex

2021-12-06 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37563: Summary: Implement days, seconds, microseconds properties of TimedeltaIndex Key: SPARK-37563 URL: https://issues.apache.org/jira/browse/SPARK-37563 Project: Spark

[jira] [Created] (SPARK-37525) Support basic operations of timedelta Series/Index

2021-12-02 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37525: Summary: Support basic operations of timedelta Series/Index Key: SPARK-37525 URL: https://issues.apache.org/jira/browse/SPARK-37525 Project: Spark Issue

[jira] [Updated] (SPARK-37512) Support TimedeltaIndex creation (from Series/Index) and TimedeltaIndex.astype

2021-12-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37512: - Summary: Support TimedeltaIndex creation (from Series/Index) and TimedeltaIndex.astype (was:

[jira] [Commented] (SPARK-37512) Support TimedeltaIndex creation given a timedelta Series/Index

2021-12-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451610#comment-17451610 ] Xinrong Meng commented on SPARK-37512: -- I am working on this. > Support TimedeltaIndex creation

[jira] [Created] (SPARK-37512) Support TimedeltaIndex creation given a timedelta Series/Index

2021-11-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37512: Summary: Support TimedeltaIndex creation given a timedelta Series/Index Key: SPARK-37512 URL: https://issues.apache.org/jira/browse/SPARK-37512 Project: Spark

[jira] [Updated] (SPARK-37511) Introduce TimedeltaIndex to pandas API on Spark

2021-11-30 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37511: - Description: Introduce TimedeltaIndex to pandas API on Spark. Properties, functions, and basic

[jira] [Created] (SPARK-37511) Introduce TimedeltaIndex to pandas API on Spark

2021-11-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37511: Summary: Introduce TimedeltaIndex to pandas API on Spark Key: SPARK-37511 URL: https://issues.apache.org/jira/browse/SPARK-37511 Project: Spark Issue Type:

[jira] [Created] (SPARK-37510) Support TimedeltaIndex in pandas API on Spark

2021-11-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37510: Summary: Support TimedeltaIndex in pandas API on Spark Key: SPARK-37510 URL: https://issues.apache.org/jira/browse/SPARK-37510 Project: Spark Issue Type:

[jira] [Updated] (SPARK-37337) Improve the API of Spark DataFrame to pandas-on-Spark DataFrame conversion

2021-11-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37337: - Description: - Undeprecate (Spark)DataFrame.to_koalas  - Deprecate

[jira] [Resolved] (SPARK-37338) Rename (Spark)DataFrame.to_pandas_on_spark to (Spark)DataFrame.pandas_api

2021-11-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37338. -- Resolution: Duplicate > Rename (Spark)DataFrame.to_pandas_on_spark to

[jira] [Updated] (SPARK-37338) Rename (Spark)DataFrame.to_pandas_on_spark to (Spark)DataFrame.pandas_api

2021-11-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37338: - Summary: Rename (Spark)DataFrame.to_pandas_on_spark to (Spark)DataFrame.pandas_api (was:

[jira] [Updated] (SPARK-37338) Rename to_pandas_on_spark to pandas_api

2021-11-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37338: - Description: Currently, (Spark)DataFrame.to_pandas_on_spark is too long to memorize and

[jira] [Created] (SPARK-37338) Rename to_pandas_on_spark to pandas_api

2021-11-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37338: Summary: Rename to_pandas_on_spark to pandas_api Key: SPARK-37338 URL: https://issues.apache.org/jira/browse/SPARK-37338 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37337) Improve the API of Spark DataFrame to pandas-on-Spark DataFrame conversion

2021-11-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37337: Summary: Improve the API of Spark DataFrame to pandas-on-Spark DataFrame conversion Key: SPARK-37337 URL: https://issues.apache.org/jira/browse/SPARK-37337 Project:

[jira] [Resolved] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37246. -- Resolution: Duplicate > Support Python 3.10 in PySpark > -- > >

[jira] [Commented] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440872#comment-17440872 ] Xinrong Meng commented on SPARK-37246: -- Oh sorry for the confusion. I was about to resolve the

[jira] [Resolved] (SPARK-37247) Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37247. -- Resolution: Invalid > Failed test_create_nan_decimal_dataframe >

[jira] [Resolved] (SPARK-37250) Failed test_capture_user_friendly_exception (pyspark.sql.tests.test_utils.UtilsTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37250. -- Resolution: Invalid > Failed test_capture_user_friendly_exception >

[jira] [Resolved] (SPARK-37248) Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37248. -- Resolution: Invalid > Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests)

[jira] [Resolved] (SPARK-37251) Failed _joinAsOf doctest

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-37251. -- Resolution: Invalid > Failed _joinAsOf doctest > > >

[jira] [Created] (SPARK-37251) Failed _joinAsOf doctest

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37251: Summary: Failed _joinAsOf doctest Key: SPARK-37251 URL: https://issues.apache.org/jira/browse/SPARK-37251 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37250) Failed test_capture_user_friendly_exception (pyspark.sql.tests.test_utils.UtilsTests)

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37250: Summary: Failed test_capture_user_friendly_exception (pyspark.sql.tests.test_utils.UtilsTests) Key: SPARK-37250 URL: https://issues.apache.org/jira/browse/SPARK-37250

[jira] [Created] (SPARK-37249) ImportError: cannot import name 'Callable' from 'collections'

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37249: Summary: ImportError: cannot import name 'Callable' from 'collections' Key: SPARK-37249 URL: https://issues.apache.org/jira/browse/SPARK-37249 Project: Spark

[jira] [Created] (SPARK-37248) Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests)

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37248: Summary: Failed test_make_date (pyspark.sql.tests.test_functions.FunctionsTests) Key: SPARK-37248 URL: https://issues.apache.org/jira/browse/SPARK-37248 Project:

[jira] [Updated] (SPARK-37247) Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests)

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37247: - Description: {code:java}   File

[jira] [Created] (SPARK-37247) Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests)

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37247: Summary: Failed test_create_nan_decimal_dataframe (pyspark.sql.tests.test_dataframe.DataFrameTests) Key: SPARK-37247 URL: https://issues.apache.org/jira/browse/SPARK-37247

[jira] [Updated] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37246: - Issue Type: Umbrella (was: Improvement) > Support Python 3.10 in PySpark >

[jira] [Updated] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-37246: - Description: Though pyarrow doesn’t support python 3.10 yet

[jira] [Created] (SPARK-37246) Support Python 3.10 in PySpark

2021-11-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-37246: Summary: Support Python 3.10 in PySpark Key: SPARK-37246 URL: https://issues.apache.org/jira/browse/SPARK-37246 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427922#comment-17427922 ] Xinrong Meng commented on SPARK-36991: -- I am working on this. > Inline type hints for

[jira] [Created] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36991: Summary: Inline type hints for spark/python/pyspark/sql/streaming.py Key: SPARK-36991 URL: https://issues.apache.org/jira/browse/SPARK-36991 Project: Spark

[jira] [Created] (SPARK-36951) Inline type hints for python/pyspark/sql/column.py

2021-10-07 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36951: Summary: Inline type hints for python/pyspark/sql/column.py Key: SPARK-36951 URL: https://issues.apache.org/jira/browse/SPARK-36951 Project: Spark Issue

[jira] [Commented] (SPARK-36951) Inline type hints for python/pyspark/sql/column.py

2021-10-07 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425794#comment-17425794 ] Xinrong Meng commented on SPARK-36951: -- I am working on this. > Inline type hints for

[jira] [Commented] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

2021-10-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425213#comment-17425213 ] Xinrong Meng commented on SPARK-36942: -- I'm working on that. > Inline type hints for

[jira] [Created] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py

2021-10-06 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36942: Summary: Inline type hints for python/pyspark/sql/readwriter.py Key: SPARK-36942 URL: https://issues.apache.org/jira/browse/SPARK-36942 Project: Spark Issue

[jira] [Updated] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

2021-10-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36940: - Description: Inline type hints for python/pyspark/sql/avro/functions.py.   Currently, we use

[jira] [Created] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

2021-10-06 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36940: Summary: Inline type hints for python/pyspark/sql/avro/functions.py Key: SPARK-36940 URL: https://issues.apache.org/jira/browse/SPARK-36940 Project: Spark

[jira] [Commented] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py

2021-10-06 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425132#comment-17425132 ] Xinrong Meng commented on SPARK-36940: -- I'm working on this. > Inline type hints for

[jira] [Commented] (SPARK-36927) Inline type hints for python/pyspark/sql/window.py

2021-10-04 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17424131#comment-17424131 ] Xinrong Meng commented on SPARK-36927: -- I'm working on this. > Inline type hints for

[jira] [Created] (SPARK-36927) Inline type hints for python/pyspark/sql/window.py

2021-10-04 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36927: Summary: Inline type hints for python/pyspark/sql/window.py Key: SPARK-36927 URL: https://issues.apache.org/jira/browse/SPARK-36927 Project: Spark Issue

[jira] [Commented] (SPARK-36910) Inline type hints for python/pyspark/sql/types.py

2021-10-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423426#comment-17423426 ] Xinrong Meng commented on SPARK-36910: -- I am working on this. > Inline type hints for

[jira] [Created] (SPARK-36910) Inline type hints for python/pyspark/sql/types.py

2021-10-01 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36910: Summary: Inline type hints for python/pyspark/sql/types.py Key: SPARK-36910 URL: https://issues.apache.org/jira/browse/SPARK-36910 Project: Spark Issue

[jira] [Updated] (SPARK-36906) Inline type hints for conf.py and observation.py in python/pyspark/sql

2021-09-30 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36906: - Description: Inline type hints for conf.py and observation.py in python/pyspark/sql to leverage

[jira] [Created] (SPARK-36906) Inline type hints for conf.py and observation.py in python/pyspark/sql

2021-09-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36906: Summary: Inline type hints for conf.py and observation.py in python/pyspark/sql Key: SPARK-36906 URL: https://issues.apache.org/jira/browse/SPARK-36906 Project:

[jira] [Created] (SPARK-36897) Replace collections.namedtuple() by typing.NamedTuple

2021-09-29 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36897: Summary: Replace collections.namedtuple() by typing.NamedTuple Key: SPARK-36897 URL: https://issues.apache.org/jira/browse/SPARK-36897 Project: Spark Issue

[jira] [Updated] (SPARK-36896) Return boolean for `dropTempView` and `dropGlobalTempView`

2021-09-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36896: - Environment: Currently`dropTempView` and `dropGlobalTempView` don't have return value, which

[jira] [Updated] (SPARK-36896) Return boolean for `dropTempView` and `dropGlobalTempView`

2021-09-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36896: - Summary: Return boolean for `dropTempView` and `dropGlobalTempView` (was: Fix returns of

[jira] [Updated] (SPARK-36896) Fix returns of functions in python/pyspark/sql/catalog.py

2021-09-29 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36896: - Summary: Fix returns of functions in python/pyspark/sql/catalog.py (was: Fix returns of

[jira] [Created] (SPARK-36896) Fix returns of functions python/pyspark/sql/catalog.py

2021-09-29 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36896: Summary: Fix returns of functions python/pyspark/sql/catalog.py Key: SPARK-36896 URL: https://issues.apache.org/jira/browse/SPARK-36896 Project: Spark Issue

[jira] [Created] (SPARK-36881) Inline type hints for python/pyspark/sql/catalog.py

2021-09-28 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36881: Summary: Inline type hints for python/pyspark/sql/catalog.py Key: SPARK-36881 URL: https://issues.apache.org/jira/browse/SPARK-36881 Project: Spark Issue

[jira] [Created] (SPARK-36880) Inline type hints for python/pyspark/sql/functions.py

2021-09-28 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36880: Summary: Inline type hints for python/pyspark/sql/functions.py Key: SPARK-36880 URL: https://issues.apache.org/jira/browse/SPARK-36880 Project: Spark Issue

[jira] [Created] (SPARK-36818) Fix filtering a Series by a boolean Series

2021-09-21 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36818: Summary: Fix filtering a Series by a boolean Series Key: SPARK-36818 URL: https://issues.apache.org/jira/browse/SPARK-36818 Project: Spark Issue Type:

[jira] [Created] (SPARK-36771) Fix `pop` of Categorical Series

2021-09-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36771: Summary: Fix `pop` of Categorical Series Key: SPARK-36771 URL: https://issues.apache.org/jira/browse/SPARK-36771 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-36771) Fix `pop` of Categorical Series

2021-09-15 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415796#comment-17415796 ] Xinrong Meng commented on SPARK-36771: -- I am working on that. > Fix `pop` of Categorical Series >

[jira] [Created] (SPARK-36769) Improve `filter` of single-indexed DataFrame

2021-09-15 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36769: Summary: Improve `filter` of single-indexed DataFrame Key: SPARK-36769 URL: https://issues.apache.org/jira/browse/SPARK-36769 Project: Spark Issue Type:

[jira] [Created] (SPARK-36748) Introduce the 'compute.isin_limit' option

2021-09-13 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36748: Summary: Introduce the 'compute.isin_limit' option Key: SPARK-36748 URL: https://issues.apache.org/jira/browse/SPARK-36748 Project: Spark Issue Type:

[jira] [Commented] (SPARK-36748) Introduce the 'compute.isin_limit' option

2021-09-13 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414629#comment-17414629 ] Xinrong Meng commented on SPARK-36748: -- I am working on that. > Introduce the 'compute.isin_limit'

[jira] [Created] (SPARK-36746) Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin

2021-09-13 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36746: Summary: Refactor _select_rows_by_iterable in iLocIndexer to use Column.isin Key: SPARK-36746 URL: https://issues.apache.org/jira/browse/SPARK-36746 Project: Spark

[jira] [Resolved] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

2021-09-13 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-36031. -- Resolution: Done > Keep same behavior with pandas for operations of series with nan >

[jira] [Commented] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412179#comment-17412179 ] Xinrong Meng commented on SPARK-36697: -- I am working on that now. > Fix dropping all columns of a

[jira] [Created] (SPARK-36697) Fix dropping all columns of a DataFrame

2021-09-08 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36697: Summary: Fix dropping all columns of a DataFrame Key: SPARK-36697 URL: https://issues.apache.org/jira/browse/SPARK-36697 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-36618) Support dropping rows of a single-indexed DataFrame

2021-09-07 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36618: - Summary: Support dropping rows of a single-indexed DataFrame (was: Support axis=0 for drop

[jira] [Commented] (SPARK-36618) Support axis=0 for drop API

2021-09-07 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411565#comment-17411565 ] Xinrong Meng commented on SPARK-36618: -- I am working on that. > Support axis=0 for drop API >

[jira] [Created] (SPARK-36655) Add `versionadded` for API added in Spark 3.3.0

2021-09-02 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36655: Summary: Add `versionadded` for API added in Spark 3.3.0 Key: SPARK-36655 URL: https://issues.apache.org/jira/browse/SPARK-36655 Project: Spark Issue Type:

[jira] [Created] (SPARK-36648) Implement __setitem__ of label-based MultiIndex

2021-09-01 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36648: Summary: Implement __setitem__ of label-based MultiIndex Key: SPARK-36648 URL: https://issues.apache.org/jira/browse/SPARK-36648 Project: Spark Issue Type:

[jira] [Updated] (SPARK-36628) Implement __getitem__ of label-based MultiIndex

2021-09-01 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36628: - Summary: Implement __getitem__ of label-based MultiIndex (was: Implement label-based

[jira] [Created] (SPARK-36628) Implement label-based MultiIndex select

2021-08-31 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36628: Summary: Implement label-based MultiIndex select Key: SPARK-36628 URL: https://issues.apache.org/jira/browse/SPARK-36628 Project: Spark Issue Type: Sub-task

<    2   3   4   5   6   7   8   9   10   >