[jira] [Updated] (SPARK-46116) Enriching PySpark Documentation with "Useful Links" Including Q Support and Mailing Lists

2023-11-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46116: Summary: Enriching PySpark Documentation with "Useful Links" Including Q Support and Mailing

[jira] [Created] (SPARK-46116) Add "Q Support" Link to PySpark Documentation Homepage

2023-11-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46116: --- Summary: Add "Q Support" Link to PySpark Documentation Homepage Key: SPARK-46116 URL: https://issues.apache.org/jira/browse/SPARK-46116 Project: Spark Issue

[jira] [Created] (SPARK-46112) Enforce usage of PySpark-specific Exceptions over built-in Python Exceptions

2023-11-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46112: --- Summary: Enforce usage of PySpark-specific Exceptions over built-in Python Exceptions Key: SPARK-46112 URL: https://issues.apache.org/jira/browse/SPARK-46112 Project:

[jira] [Created] (SPARK-46111) Add copyright to the PySpark official documentation.

2023-11-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46111: --- Summary: Add copyright to the PySpark official documentation. Key: SPARK-46111 URL: https://issues.apache.org/jira/browse/SPARK-46111 Project: Spark Issue

[jira] [Created] (SPARK-46087) Sync PySpark dependencies in docs and dev requirements

2023-11-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46087: --- Summary: Sync PySpark dependencies in docs and dev requirements Key: SPARK-46087 URL: https://issues.apache.org/jira/browse/SPARK-46087 Project: Spark Issue

[jira] [Updated] (SPARK-46016) Fix pandas API support list properly

2023-11-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46016: Summary: Fix pandas API support list properly (was: Correct Supported pandas API list) > Fix

[jira] [Updated] (SPARK-46016) Correct Supported pandas API list

2023-11-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46016: Summary: Correct Supported pandas API list (was: Fix the script for Supported pandas API to work

[jira] [Updated] (SPARK-46084) Refactor data type casting operation for Categorical type.

2023-11-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46084: Description: Using official API for better performance and readability. (was: Using official API

[jira] [Created] (SPARK-46084) Refactor data type casting operation for Categorical type.

2023-11-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46084: --- Summary: Refactor data type casting operation for Categorical type. Key: SPARK-46084 URL: https://issues.apache.org/jira/browse/SPARK-46084 Project: Spark

[jira] [Created] (SPARK-46065) Refactor `(DataFrame|Series).factorize()` to use `create_map`.

2023-11-22 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46065: --- Summary: Refactor `(DataFrame|Series).factorize()` to use `create_map`. Key: SPARK-46065 URL: https://issues.apache.org/jira/browse/SPARK-46065 Project: Spark

[jira] [Created] (SPARK-46045) Add individual categories for `Options and settings` to API reference

2023-11-21 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46045: --- Summary: Add individual categories for `Options and settings` to API reference Key: SPARK-46045 URL: https://issues.apache.org/jira/browse/SPARK-46045 Project: Spark

[jira] [Created] (SPARK-46022) Remove deprecated functions APIs from documents

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46022: --- Summary: Remove deprecated functions APIs from documents Key: SPARK-46022 URL: https://issues.apache.org/jira/browse/SPARK-46022 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46017) PySpark doc build doesn't work properly on Mac

2023-11-20 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46017: Summary: PySpark doc build doesn't work properly on Mac (was: PySpark doc build doesn't work

[jira] [Updated] (SPARK-46017) PySpark doc build doesn't work properly on Mac

2023-11-20 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46017: Description: PySpark doc build is working properly on GitHub CI, but doesn't work properly on

[jira] [Updated] (SPARK-46017) PySpark doc build doesn't work properly on local environment

2023-11-20 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-46017: Summary: PySpark doc build doesn't work properly on local environment (was: PySpark doc build

[jira] [Created] (SPARK-46017) PySpark doc build doesn't work properly on local env

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46017: --- Summary: PySpark doc build doesn't work properly on local env Key: SPARK-46017 URL: https://issues.apache.org/jira/browse/SPARK-46017 Project: Spark Issue

[jira] [Created] (SPARK-46016) Fix the script for Supported pandas API to work properly

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46016: --- Summary: Fix the script for Supported pandas API to work properly Key: SPARK-46016 URL: https://issues.apache.org/jira/browse/SPARK-46016 Project: Spark Issue

[jira] [Created] (SPARK-46015) Fix broken link for Koalas issues

2023-11-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46015: --- Summary: Fix broken link for Koalas issues Key: SPARK-46015 URL: https://issues.apache.org/jira/browse/SPARK-46015 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45997) Remove deprecated APIs from legacy Koalas

2023-11-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45997: Summary: Remove deprecated APIs from legacy Koalas (was: Remove deprecated Koalas signatures) >

[jira] [Updated] (SPARK-45997) Remove deprecated Koalas signatures

2023-11-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45997: Description: We should remove the deprecated Koalas signatures to cleanup the API surface. (was:

[jira] [Updated] (SPARK-45997) Remove deprecated Koalas signatures

2023-11-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45997: Summary: Remove deprecated Koalas signatures (was: Update Koalas Migration Guide) > Remove

[jira] [Created] (SPARK-45997) Update Koalas Migration Guide

2023-11-19 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45997: --- Summary: Update Koalas Migration Guide Key: SPARK-45997 URL: https://issues.apache.org/jira/browse/SPARK-45997 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45966) Add missing methods for API reference.

2023-11-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45966: --- Summary: Add missing methods for API reference. Key: SPARK-45966 URL: https://issues.apache.org/jira/browse/SPARK-45966 Project: Spark Issue Type: Bug

[jira] [Reopened] (SPARK-45551) Enhance PySpark testing utils

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-45551: - > Enhance PySpark testing utils > - > > Key:

[jira] [Resolved] (SPARK-45551) Enhance PySpark testing utils

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-45551. - Resolution: Fixed > Enhance PySpark testing utils > - > >

[jira] [Updated] (SPARK-45913) Make the internal attributes private from PySpark errors.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Summary: Make the internal attributes private from PySpark errors. (was: Make the internal

[jira] [Updated] (SPARK-45913) Make the internal attributes private from `PySparkException`.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Summary: Make the internal attributes private from `PySparkException`. (was: Hide the internal

[jira] [Updated] (SPARK-45913) Hide the internal attributes from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Description: There are some APIs from PySparkException are exposed to user space which should

[jira] [Updated] (SPARK-45913) Hide the internal attributes from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Summary: Hide the internal attributes from PySparkException. (was: Hide the internal APIs from

[jira] [Updated] (SPARK-45913) Hide the internal APIs from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Description: There are some APIs are exposed to user space which should have not been as below:

[jira] [Updated] (SPARK-45913) Hide the internal APIs from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45913: Description: There are some APIs from PySparkException are exposed to user space which should

[jira] [Created] (SPARK-45913) Hide the internal APIs from PySparkException.

2023-11-13 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45913: --- Summary: Hide the internal APIs from PySparkException. Key: SPARK-45913 URL: https://issues.apache.org/jira/browse/SPARK-45913 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45812) Upgrade Pandas to 2.1.2

2023-11-06 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45812: --- Summary: Upgrade Pandas to 2.1.2 Key: SPARK-45812 URL: https://issues.apache.org/jira/browse/SPARK-45812 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-45718) Remove remaining deprecated Pandas APIs from Spark 3.4.0

2023-10-30 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45718: --- Summary: Remove remaining deprecated Pandas APIs from Spark 3.4.0 Key: SPARK-45718 URL: https://issues.apache.org/jira/browse/SPARK-45718 Project: Spark Issue

[jira] [Reopened] (SPARK-45673) Enhancing clarity and usability of PySpark error messages

2023-10-29 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-45673: - > Enhancing clarity and usability of PySpark error messages >

[jira] [Resolved] (SPARK-45673) Enhancing clarity and usability of PySpark error messages

2023-10-29 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-45673. - Resolution: Fixed > Enhancing clarity and usability of PySpark error messages >

[jira] [Updated] (SPARK-45674) Improve error message for JVM-dependent attributes on Spark Connect.

2023-10-26 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45674: Summary: Improve error message for JVM-dependent attributes on Spark Connect. (was: Improve

[jira] [Created] (SPARK-45674) Improve `NOT_IMPLEMENTED` error for `sparkContext` on Spark Connect.

2023-10-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45674: --- Summary: Improve `NOT_IMPLEMENTED` error for `sparkContext` on Spark Connect. Key: SPARK-45674 URL: https://issues.apache.org/jira/browse/SPARK-45674 Project: Spark

[jira] [Created] (SPARK-45673) Enhancing clarity and usability of PySpark error messages

2023-10-26 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45673: --- Summary: Enhancing clarity and usability of PySpark error messages Key: SPARK-45673 URL: https://issues.apache.org/jira/browse/SPARK-45673 Project: Spark

[jira] [Updated] (SPARK-45551) Enhance PySpark testing utils

2023-10-23 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45551: Summary: Enhance PySpark testing utils (was: Enchance PySpark testing utils) > Enhance PySpark

[jira] [Created] (SPARK-45635) Cleanup unused import for PySpark testing

2023-10-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45635: --- Summary: Cleanup unused import for PySpark testing Key: SPARK-45635 URL: https://issues.apache.org/jira/browse/SPARK-45635 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45634) Remove `get_dtype_counts` from Pandas API on Spark

2023-10-23 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45634: --- Summary: Remove `get_dtype_counts` from Pandas API on Spark Key: SPARK-45634 URL: https://issues.apache.org/jira/browse/SPARK-45634 Project: Spark Issue Type:

[jira] [Created] (SPARK-45566) Introduce Pandas-like testing utils for Pandas API on Spark

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45566: --- Summary: Introduce Pandas-like testing utils for Pandas API on Spark Key: SPARK-45566 URL: https://issues.apache.org/jira/browse/SPARK-45566 Project: Spark

[jira] [Updated] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45553: Description: We will add new APIs for DataFrame, Series and Index separately, and we should

[jira] [Updated] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45553: Summary: Deprecate assertPandasOnSparkEqual (was: Introduce flexible parameters to

[jira] [Created] (SPARK-45555) Returning a debuggable object for failed assertion

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-4: --- Summary: Returning a debuggable object for failed assertion Key: SPARK-4 URL: https://issues.apache.org/jira/browse/SPARK-4 Project: Spark Issue Type:

[jira] [Created] (SPARK-45554) Introduce flexible parameter to assertSchemaEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45554: --- Summary: Introduce flexible parameter to assertSchemaEqual Key: SPARK-45554 URL: https://issues.apache.org/jira/browse/SPARK-45554 Project: Spark Issue Type:

[jira] [Created] (SPARK-45553) Introduce flexible parameters to assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45553: --- Summary: Introduce flexible parameters to assertPandasOnSparkEqual Key: SPARK-45553 URL: https://issues.apache.org/jira/browse/SPARK-45553 Project: Spark

[jira] [Created] (SPARK-45552) Introduce flexible parameters to assertDataFrameEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45552: --- Summary: Introduce flexible parameters to assertDataFrameEqual Key: SPARK-45552 URL: https://issues.apache.org/jira/browse/SPARK-45552 Project: Spark Issue

[jira] [Created] (SPARK-45551) Enchance PySpark testing utils

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45551: --- Summary: Enchance PySpark testing utils Key: SPARK-45551 URL: https://issues.apache.org/jira/browse/SPARK-45551 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-45550) Remove deprecated APIs from Pandas API on Spark

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45550: Summary: Remove deprecated APIs from Pandas API on Spark (was: Remove to_spark_io) > Remove

[jira] [Updated] (SPARK-45550) Remove to_spark_io

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45550: Summary: Remove to_spark_io (was: Remove deprecated PySpark APIs) > Remove to_spark_io >

[jira] [Created] (SPARK-45550) Remove deprecated PySpark APIs

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45550: --- Summary: Remove deprecated PySpark APIs Key: SPARK-45550 URL: https://issues.apache.org/jira/browse/SPARK-45550 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45476) Raise exception directly instead of calling `resolveColumnsByPosition`

2023-10-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45476: --- Summary: Raise exception directly instead of calling `resolveColumnsByPosition` Key: SPARK-45476 URL: https://issues.apache.org/jira/browse/SPARK-45476 Project: Spark

[jira] [Updated] (SPARK-43656) Enable NumPy compat tests

2023-10-05 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43656: Summary: Enable NumPy compat tests (was: Fix pyspark.sql.column._to_java_column to accept

[jira] [Resolved] (SPARK-44101) Support pandas 2

2023-09-27 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-44101. - Resolution: Fixed > Support pandas 2 > > > Key: SPARK-44101 >

[jira] [Updated] (SPARK-45308) Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.

2023-09-25 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45308: Summary: Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.

[jira] [Created] (SPARK-45308) Enable`GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0.

2023-09-25 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45308: --- Summary: Enable`GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0. Key: SPARK-45308 URL: https://issues.apache.org/jira/browse/SPARK-45308

[jira] [Updated] (SPARK-43877) Fix behavior difference for compare binary functions.

2023-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43877: Epic Link: SPARK-39375 > Fix behavior difference for compare binary functions. >

[jira] [Updated] (SPARK-43877) Fix behavior difference for compare binary functions.

2023-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43877: Parent: (was: SPARK-42497) Issue Type: Improvement (was: Sub-task) > Fix behavior

[jira] [Resolved] (SPARK-43623) Enable DefaultIndexParityTests.test_index_distributed_sequence_cleanup.

2023-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43623. - Resolution: Duplicate > Enable DefaultIndexParityTests.test_index_distributed_sequence_cleanup.

[jira] [Resolved] (SPARK-43159) Refine `column_op` to use lambda function instead of Column API.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43159. - Resolution: Won't Fix > Refine `column_op` to use lambda function instead of Column API. >

[jira] [Updated] (SPARK-42965) metadata mismatch for StructField when running some tests.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42965: Parent: (was: SPARK-42497) Issue Type: Improvement (was: Sub-task) > metadata

[jira] [Updated] (SPARK-42965) metadata mismatch for StructField when running some tests.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42965: Epic Link: SPARK-39375 > metadata mismatch for StructField when running some tests. >

[jira] [Updated] (SPARK-45267) Change the default value for `numeric_only`.

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45267: Summary: Change the default value for `numeric_only`. (was: Changed the default value for

[jira] [Created] (SPARK-45267) Changed the default value for `numeric_only`.

2023-09-21 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45267: --- Summary: Changed the default value for `numeric_only`. Key: SPARK-45267 URL: https://issues.apache.org/jira/browse/SPARK-45267 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-45264) Configurable error when generating Python docs

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767764#comment-17767764 ] Haejoon Lee edited comment on SPARK-45264 at 9/22/23 12:28 AM: --- Currently

[jira] [Commented] (SPARK-45264) Configurable error when generating Python docs

2023-09-21 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767764#comment-17767764 ] Haejoon Lee commented on SPARK-45264: - Currently the PySpark documentation build requires installing

[jira] [Created] (SPARK-45247) Upgrade Pandas to 2.1.1

2023-09-21 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45247: --- Summary: Upgrade Pandas to 2.1.1 Key: SPARK-45247 URL: https://issues.apache.org/jira/browse/SPARK-45247 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-45246) Encourage using latest jinja2 other than documentation build

2023-09-20 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45246: --- Summary: Encourage using latest jinja2 other than documentation build Key: SPARK-45246 URL: https://issues.apache.org/jira/browse/SPARK-45246 Project: Spark

[jira] [Updated] (SPARK-45228) Update `test_axis_on_dataframe` when Pandas regression is fixed

2023-09-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45228: Summary: Update `test_axis_on_dataframe` when Pandas regression is fixed (was: Restore

[jira] [Created] (SPARK-45228) Restore `test_axis_on_dataframe` in normal state when Pandas regression is fixed

2023-09-19 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45228: --- Summary: Restore `test_axis_on_dataframe` in normal state when Pandas regression is fixed Key: SPARK-45228 URL: https://issues.apache.org/jira/browse/SPARK-45228

[jira] [Updated] (SPARK-43432) Fix `min_periods` for Rolling to work same as pandas

2023-09-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-43432: Parent: (was: SPARK-44101) Issue Type: Improvement (was: Sub-task) > Fix

[jira] [Reopened] (SPARK-43433) Match `GroupBy.nth` behavior with new pandas behavior

2023-09-19 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-43433: - > Match `GroupBy.nth` behavior with new pandas behavior >

[jira] [Commented] (SPARK-44033) Support list-like for binary ops

2023-09-17 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766170#comment-17766170 ] Haejoon Lee commented on SPARK-44033: - No worries! Let me take a look soon. Thanks for working on

[jira] [Resolved] (SPARK-45185) Ignore type check for preventing unexpected linter failure

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-45185. - Resolution: Won't Fix > Ignore type check for preventing unexpected linter failure >

[jira] [Resolved] (SPARK-43630) Implement `localCheckpoint` for Spark Connect DataFrame

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43630. - Resolution: Won't Fix We can't support RDD-dependent feature in Spark Connect. > Implement

[jira] [Reopened] (SPARK-43628) Enable SparkContext-related tests with Spark Connect

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reopened SPARK-43628: - > Enable SparkContext-related tests with Spark Connect >

[jira] [Comment Edited] (SPARK-43628) Enable SparkContext-related tests with Spark Connect

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17765930#comment-17765930 ] Haejoon Lee edited comment on SPARK-43628 at 9/16/23 9:14 AM: -- We can't

[jira] (SPARK-43628) Enable SparkContext-related tests with Spark Connect

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43628 ] Haejoon Lee deleted comment on SPARK-43628: - was (Author: itholic): We can't support SparkContext tests in Spark Connect. > Enable SparkContext-related tests with Spark Connect >

[jira] [Resolved] (SPARK-43628) Enable SparkContext-related tests with Spark Connect

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43628. - Resolution: Won't Fix We can't support SparkConnect tests in Spark Connect. > Enable

[jira] [Resolved] (SPARK-43045) Clean-up remaining mypy failure

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43045. - Resolution: Won't Do I believe the current status good enough to check the static analytics for

[jira] [Resolved] (SPARK-43003) Enable doctest

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43003. - Resolution: Works for Me > Enable doctest > -- > > Key: SPARK-43003

[jira] [Resolved] (SPARK-44492) Resolve remaining AnalysisException

2023-09-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-44492. - Fix Version/s: 4.0.0 Target Version/s: 4.0.0 Resolution: Fixed > Resolve

[jira] [Created] (SPARK-45185) Ignore type check for preventing unexpected linter failure

2023-09-15 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45185: --- Summary: Ignore type check for preventing unexpected linter failure Key: SPARK-45185 URL: https://issues.apache.org/jira/browse/SPARK-45185 Project: Spark

[jira] [Created] (SPARK-45180) Remove boolean inputs for inclusive from Series.between

2023-09-15 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45180: --- Summary: Remove boolean inputs for inclusive from Series.between Key: SPARK-45180 URL: https://issues.apache.org/jira/browse/SPARK-45180 Project: Spark Issue

[jira] [Resolved] (SPARK-43811) Enable DataFrameTests.test_reindex for pandas 2.0.0.

2023-09-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43811. - Resolution: Fixed > Enable DataFrameTests.test_reindex for pandas 2.0.0. >

[jira] [Resolved] (SPARK-44276) Match behavior with pandas for `SeriesStringTests.test_string_replace`

2023-09-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-44276. - Resolution: Fixed > Match behavior with pandas for `SeriesStringTests.test_string_replace` >

[jira] [Resolved] (SPARK-43644) Enable DatetimeIndexTests.test_indexer_between_time for pandas 2.0.0.

2023-09-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43644. - Resolution: Fixed > Enable DatetimeIndexTests.test_indexer_between_time for pandas 2.0.0. >

[jira] [Resolved] (SPARK-43433) Match `GroupBy.nth` behavior with new pandas behavior

2023-09-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43433. - Resolution: Fixed > Match `GroupBy.nth` behavior with new pandas behavior >

[jira] [Resolved] (SPARK-43291) Generate proper warning on different behavior with numeric_only

2023-09-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43291. - Resolution: Won't Fix We should match the behavior w/ Pandas instead of warning. > Generate

[jira] [Resolved] (SPARK-43282) Investigate DataFrame.sort_values with pandas behavior.

2023-09-15 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43282. - Resolution: Won't Fix > Investigate DataFrame.sort_values with pandas behavior. >

[jira] [Resolved] (SPARK-43271) Match behavior with DataFrame.reindex with specifying `index`.

2023-09-14 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43271. - Resolution: Fixed > Match behavior with DataFrame.reindex with specifying `index`. >

[jira] [Created] (SPARK-45177) Remove `col_space` parameter from `DataFrame.to_latex`

2023-09-14 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45177: --- Summary: Remove `col_space` parameter from `DataFrame.to_latex` Key: SPARK-45177 URL: https://issues.apache.org/jira/browse/SPARK-45177 Project: Spark Issue

[jira] [Updated] (SPARK-45165) Remove `inplace` parameter from `Categorical` APIs

2023-09-14 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45165: Summary: Remove `inplace` parameter from `Categorical` APIs (was: Remove `inplace` parameter

[jira] [Created] (SPARK-45165) Remove `inplace` parameter from `CategoricalIndex` APIs

2023-09-14 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45165: --- Summary: Remove `inplace` parameter from `CategoricalIndex` APIs Key: SPARK-45165 URL: https://issues.apache.org/jira/browse/SPARK-45165 Project: Spark Issue

[jira] [Updated] (SPARK-45164) Remove deprecated Index APIs

2023-09-14 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45164: Description: We should remove the deprecated Index APIs to match the behavior with Pandas 2.0.0

[jira] [Created] (SPARK-45164) Remove deprecated Index APIs

2023-09-14 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45164: --- Summary: Remove deprecated Index APIs Key: SPARK-45164 URL: https://issues.apache.org/jira/browse/SPARK-45164 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-43335) Migrate remaining Connect/SparkSQL errors into error classes

2023-09-14 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43335. - Resolution: Fixed > Migrate remaining Connect/SparkSQL errors into error classes >

[jira] [Resolved] (SPARK-43303) Migrate NotImplementedError into PySparkNotImplementedError

2023-09-14 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43303. - Resolution: Fixed > Migrate NotImplementedError into PySparkNotImplementedError >

<    1   2   3   4   5   6   7   8   9   10   >