[jira] [Created] (SPARK-44200) Support TABLE argument parser rule for TableValuedFunction

2023-06-26 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-44200: - Summary: Support TABLE argument parser rule for TableValuedFunction Key: SPARK-44200 URL: https://issues.apache.org/jira/browse/SPARK-44200 Project: Spark

[jira] [Resolved] (SPARK-43804) Test on nested structs support in Pandas UDF

2023-05-31 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-43804. --- Assignee: Xinrong Meng Resolution: Fixed Issue resolved by pull request 41320

[jira] [Created] (SPARK-43817) Support UserDefinedType in creaetDataFrame from pandas DataFrame and toPandas

2023-05-26 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43817: - Summary: Support UserDefinedType in creaetDataFrame from pandas DataFrame and toPandas Key: SPARK-43817 URL: https://issues.apache.org/jira/browse/SPARK-43817

[jira] [Created] (SPARK-43759) Expose TimestampNTZType in pyspark.sql.types

2023-05-23 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43759: - Summary: Expose TimestampNTZType in pyspark.sql.types Key: SPARK-43759 URL: https://issues.apache.org/jira/browse/SPARK-43759 Project: Spark Issue Type:

[jira] [Created] (SPARK-43531) Enable more parity tests for Pandas UDFs.

2023-05-16 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43531: - Summary: Enable more parity tests for Pandas UDFs. Key: SPARK-43531 URL: https://issues.apache.org/jira/browse/SPARK-43531 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-43528) Support duplicated field names in createDataFrame with pandas DataFrame.

2023-05-16 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43528: - Summary: Support duplicated field names in createDataFrame with pandas DataFrame. Key: SPARK-43528 URL: https://issues.apache.org/jira/browse/SPARK-43528 Project:

[jira] [Created] (SPARK-43473) Support struct type in createDataFrame from pandas DataFrame

2023-05-11 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43473: - Summary: Support struct type in createDataFrame from pandas DataFrame Key: SPARK-43473 URL: https://issues.apache.org/jira/browse/SPARK-43473 Project: Spark

[jira] [Created] (SPARK-43363) Remove a workaround for pandas categorical type for pyarrow

2023-05-03 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43363: - Summary: Remove a workaround for pandas categorical type for pyarrow Key: SPARK-43363 URL: https://issues.apache.org/jira/browse/SPARK-43363 Project: Spark

[jira] [Created] (SPARK-43323) DataFrame.toPandas with Arrow enabled should handle exceptions properly

2023-04-28 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43323: - Summary: DataFrame.toPandas with Arrow enabled should handle exceptions properly Key: SPARK-43323 URL: https://issues.apache.org/jira/browse/SPARK-43323 Project:

[jira] [Created] (SPARK-43153) Skip Spark execution when the dataframe is local.

2023-04-15 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43153: - Summary: Skip Spark execution when the dataframe is local. Key: SPARK-43153 URL: https://issues.apache.org/jira/browse/SPARK-43153 Project: Spark Issue

[jira] [Created] (SPARK-43146) Implement eager evaluation.

2023-04-14 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43146: - Summary: Implement eager evaluation. Key: SPARK-43146 URL: https://issues.apache.org/jira/browse/SPARK-43146 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-43115) Split pyspark-pandas-connect from pyspark-connect module.

2023-04-12 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43115: - Summary: Split pyspark-pandas-connect from pyspark-connect module. Key: SPARK-43115 URL: https://issues.apache.org/jira/browse/SPARK-43115 Project: Spark

[jira] [Resolved] (SPARK-42437) Pyspark catalog.cacheTable allow to specify storage level Connect add support Storagelevel

2023-04-12 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-42437. --- Fix Version/s: 3.5.0 Assignee: Khalid Mammadov Resolution: Fixed Issue

[jira] [Updated] (SPARK-43062) Add options to lint-python to run each test separately

2023-04-07 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-43062: -- Priority: Minor (was: Major) > Add options to lint-python to run each test separately >

[jira] [Created] (SPARK-43062) Add options to lint-python to run each test separately

2023-04-07 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43062: - Summary: Add options to lint-python to run each test separately Key: SPARK-43062 URL: https://issues.apache.org/jira/browse/SPARK-43062 Project: Spark

[jira] [Created] (SPARK-43055) createDataFrame should support duplicated nested field names

2023-04-06 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-43055: - Summary: createDataFrame should support duplicated nested field names Key: SPARK-43055 URL: https://issues.apache.org/jira/browse/SPARK-43055 Project: Spark

[jira] [Created] (SPARK-42998) Fix DataFrame.collect with null struct.

2023-03-31 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42998: - Summary: Fix DataFrame.collect with null struct. Key: SPARK-42998 URL: https://issues.apache.org/jira/browse/SPARK-42998 Project: Spark Issue Type:

[jira] [Created] (SPARK-42985) Fix createDataFrame from pandas to respect session timezone.

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42985: - Summary: Fix createDataFrame from pandas to respect session timezone. Key: SPARK-42985 URL: https://issues.apache.org/jira/browse/SPARK-42985 Project: Spark

[jira] [Created] (SPARK-42984) Fix test_createDataFrame_with_single_data_type.

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42984: - Summary: Fix test_createDataFrame_with_single_data_type. Key: SPARK-42984 URL: https://issues.apache.org/jira/browse/SPARK-42984 Project: Spark Issue

[jira] [Created] (SPARK-42983) Fix the error message of createDataFrame from np.array(0)

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42983: - Summary: Fix the error message of createDataFrame from np.array(0) Key: SPARK-42983 URL: https://issues.apache.org/jira/browse/SPARK-42983 Project: Spark

[jira] [Created] (SPARK-42982) Fix createDataFrame from pandas with map type

2023-03-30 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42982: - Summary: Fix createDataFrame from pandas with map type Key: SPARK-42982 URL: https://issues.apache.org/jira/browse/SPARK-42982 Project: Spark Issue Type:

[jira] [Created] (SPARK-42970) Reuse pyspark.sql.tests.test_arrow test cases

2023-03-29 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42970: - Summary: Reuse pyspark.sql.tests.test_arrow test cases Key: SPARK-42970 URL: https://issues.apache.org/jira/browse/SPARK-42970 Project: Spark Issue Type:

[jira] [Created] (SPARK-42969) Fix the comparison the result with Arrow optimization enabled/disabled.

2023-03-29 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42969: - Summary: Fix the comparison the result with Arrow optimization enabled/disabled. Key: SPARK-42969 URL: https://issues.apache.org/jira/browse/SPARK-42969 Project:

[jira] [Created] (SPARK-42920) Python UDF with UDT

2023-03-24 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42920: - Summary: Python UDF with UDT Key: SPARK-42920 URL: https://issues.apache.org/jira/browse/SPARK-42920 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-42911) Introduce more basic exceptions.

2023-03-23 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42911: - Summary: Introduce more basic exceptions. Key: SPARK-42911 URL: https://issues.apache.org/jira/browse/SPARK-42911 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-42900) Fix createDataFrame to respect both type inference and column names.

2023-03-22 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42900: - Summary: Fix createDataFrame to respect both type inference and column names. Key: SPARK-42900 URL: https://issues.apache.org/jira/browse/SPARK-42900 Project:

[jira] [Updated] (SPARK-42899) DataFrame.to(schema) fails when it contains non-nullable nested field in nullable field

2023-03-22 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42899: -- Summary: DataFrame.to(schema) fails when it contains non-nullable nested field in nullable

[jira] [Updated] (SPARK-42899) DataFrame.to(schema) fails with the schema of itself.

2023-03-22 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42899: -- Description: {{DataFrame.to(schema)}} fails when it contains non-nullable nested field in

[jira] [Created] (SPARK-42899) DataFrame.to(schema) fails with the schema of itself.

2023-03-22 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42899: - Summary: DataFrame.to(schema) fails with the schema of itself. Key: SPARK-42899 URL: https://issues.apache.org/jira/browse/SPARK-42899 Project: Spark

[jira] [Created] (SPARK-42889) Implement cache, persist, unpersist, and storageLevel

2023-03-21 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42889: - Summary: Implement cache, persist, unpersist, and storageLevel Key: SPARK-42889 URL: https://issues.apache.org/jira/browse/SPARK-42889 Project: Spark

[jira] [Created] (SPARK-42875) Fix toPandas to handle timezone and map types properly.

2023-03-20 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42875: - Summary: Fix toPandas to handle timezone and map types properly. Key: SPARK-42875 URL: https://issues.apache.org/jira/browse/SPARK-42875 Project: Spark

[jira] [Created] (SPARK-42848) Implement DataFrame.registerTempTable

2023-03-17 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42848: - Summary: Implement DataFrame.registerTempTable Key: SPARK-42848 URL: https://issues.apache.org/jira/browse/SPARK-42848 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-41922) Implement DataFrame `semanticHash`

2023-03-17 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-41922. --- Resolution: Duplicate > Implement DataFrame `semanticHash` >

[jira] [Created] (SPARK-42818) Implement DataFrameReader/Writer.jdbc

2023-03-15 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42818: - Summary: Implement DataFrameReader/Writer.jdbc Key: SPARK-42818 URL: https://issues.apache.org/jira/browse/SPARK-42818 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-42733) df.write.format().save() should support calling with no path or table name

2023-03-09 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42733: -- Parent: SPARK-41284 Issue Type: Sub-task (was: Bug) > df.write.format().save()

[jira] [Created] (SPARK-42705) SparkSession.sql doesn't return values from commands.

2023-03-07 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42705: - Summary: SparkSession.sql doesn't return values from commands. Key: SPARK-42705 URL: https://issues.apache.org/jira/browse/SPARK-42705 Project: Spark

[jira] [Resolved] (SPARK-41843) Implement SparkSession.udf

2023-03-01 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-41843. --- Fix Version/s: 3.4.0 Resolution: Fixed > Implement SparkSession.udf >

[jira] [Updated] (SPARK-42624) Reorganize imports in test_functions

2023-02-28 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42624: -- Component/s: PySpark (was: SQL) > Reorganize imports in test_functions >

[jira] [Created] (SPARK-42624) Reorganize imports in test_functions

2023-02-28 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42624: - Summary: Reorganize imports in test_functions Key: SPARK-42624 URL: https://issues.apache.org/jira/browse/SPARK-42624 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-42612) Enable more parity tests related to functions

2023-02-27 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42612: - Summary: Enable more parity tests related to functions Key: SPARK-42612 URL: https://issues.apache.org/jira/browse/SPARK-42612 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-42510) Implement `DataFrame.mapInPandas`

2023-02-27 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-42510. --- Fix Version/s: 3.4.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Created] (SPARK-42574) DataFrame.toPandas should handle duplicated column names

2023-02-24 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42574: - Summary: DataFrame.toPandas should handle duplicated column names Key: SPARK-42574 URL: https://issues.apache.org/jira/browse/SPARK-42574 Project: Spark

[jira] [Created] (SPARK-42570) Fix DataFrameReader to use the default source

2023-02-24 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42570: - Summary: Fix DataFrameReader to use the default source Key: SPARK-42570 URL: https://issues.apache.org/jira/browse/SPARK-42570 Project: Spark Issue Type:

[jira] [Created] (SPARK-42568) SparkConnectStreamHandler should manage configs properly while creating plans.

2023-02-24 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42568: - Summary: SparkConnectStreamHandler should manage configs properly while creating plans. Key: SPARK-42568 URL: https://issues.apache.org/jira/browse/SPARK-42568

[jira] [Created] (SPARK-42522) Fix DataFrameWriterV2 to find the default source

2023-02-21 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42522: - Summary: Fix DataFrameWriterV2 to find the default source Key: SPARK-42522 URL: https://issues.apache.org/jira/browse/SPARK-42522 Project: Spark Issue

[jira] [Commented] (SPARK-41901) Parity in String representation of Column

2023-02-17 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690652#comment-17690652 ] Takuya Ueshin commented on SPARK-41901: --- For the first case, {{{}ACOSH{}}}, {{{}ASINH{}}}, and

[jira] [Created] (SPARK-42458) createDataFrame should support DDL string as schema

2023-02-15 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42458: - Summary: createDataFrame should support DDL string as schema Key: SPARK-42458 URL: https://issues.apache.org/jira/browse/SPARK-42458 Project: Spark Issue

[jira] [Updated] (SPARK-42426) insertInto fails when the column names are different from the table columns

2023-02-13 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42426: -- Summary: insertInto fails when the column names are different from the table columns (was:

[jira] [Created] (SPARK-42426) insertInto doesn't insert when the column names are different from the table columns

2023-02-13 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42426: - Summary: insertInto doesn't insert when the column names are different from the table columns Key: SPARK-42426 URL: https://issues.apache.org/jira/browse/SPARK-42426

[jira] [Updated] (SPARK-42426) insertInto doesn't insert when the column names are different from the table columns

2023-02-13 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42426: -- Description: {noformat} File "/.../python/pyspark/sql/connect/readwriter.py", line 518, in

[jira] [Updated] (SPARK-41870) Handle duplicate columns in `createDataFrame`

2023-02-13 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-41870: -- Attachment: (was: session.py) > Handle duplicate columns in `createDataFrame` >

[jira] [Updated] (SPARK-41870) Handle duplicate columns in `createDataFrame`

2023-02-13 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-41870: -- Attachment: session.py > Handle duplicate columns in `createDataFrame` >

[jira] [Resolved] (SPARK-42265) DataFrame.createTempView - SparkConnectGrpcException: requirement failed

2023-02-13 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-42265. --- Assignee: Takuya Ueshin Resolution: Fixed Issue resolved by pull request 39968

[jira] [Resolved] (SPARK-41820) DataFrame.createOrReplaceGlobalTempView - SparkConnectException: requirement failed

2023-02-13 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-41820. --- Assignee: Takuya Ueshin Resolution: Fixed Issue resolved by pull request 39968

[jira] [Created] (SPARK-42402) Support parameterized SQL by sql()

2023-02-10 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42402: - Summary: Support parameterized SQL by sql() Key: SPARK-42402 URL: https://issues.apache.org/jira/browse/SPARK-42402 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-41820) DataFrame.createOrReplaceGlobalTempView - SparkConnectException: requirement failed

2023-02-10 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-41820: -- Description: {code:java} >>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")],

[jira] [Updated] (SPARK-42017) df["bad_key"] does not raise AnalysisException

2023-02-09 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42017: -- Parent Issue: SPARK-41282 (was: SPARK-42006) > df["bad_key"] does not raise

[jira] [Updated] (SPARK-42017) df["bad_key"] does not raise AnalysisException

2023-02-06 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42017: -- Summary: df["bad_key"] does not raise AnalysisException (was: Different error type

[jira] [Updated] (SPARK-42338) Different exception in DataFrame.sample

2023-02-03 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42338: -- Environment: (was: It raises {{SparkConnectGrpcException}} instead of

[jira] [Updated] (SPARK-42338) Different exception in DataFrame.sample

2023-02-03 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-42338: -- Description: It raises {{SparkConnectGrpcException}} instead of {{IllegalArgumentException}}.

[jira] [Created] (SPARK-42342) Introduce base hierarchy to exceptions.

2023-02-03 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42342: - Summary: Introduce base hierarchy to exceptions. Key: SPARK-42342 URL: https://issues.apache.org/jira/browse/SPARK-42342 Project: Spark Issue Type:

[jira] [Created] (SPARK-42340) Implement GroupedData.applyInPandas

2023-02-03 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42340: - Summary: Implement GroupedData.applyInPandas Key: SPARK-42340 URL: https://issues.apache.org/jira/browse/SPARK-42340 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-42338) Different exception in DataFrame.sample

2023-02-03 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42338: - Summary: Different exception in DataFrame.sample Key: SPARK-42338 URL: https://issues.apache.org/jira/browse/SPARK-42338 Project: Spark Issue Type:

[jira] [Commented] (SPARK-42017) Different error type AnalysisException vs SparkConnectAnalysisException

2023-02-02 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17683594#comment-17683594 ] Takuya Ueshin commented on SPARK-42017: --- The error class hierarchy is one of the issues, but the

[jira] [Created] (SPARK-42295) Tear down the test cleanly

2023-02-02 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-42295: - Summary: Tear down the test cleanly Key: SPARK-42295 URL: https://issues.apache.org/jira/browse/SPARK-42295 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-41778) Add an alias "reduce" to ArrayAggregate

2022-12-29 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-41778: - Summary: Add an alias "reduce" to ArrayAggregate Key: SPARK-41778 URL: https://issues.apache.org/jira/browse/SPARK-41778 Project: Spark Issue Type:

[jira] [Created] (SPARK-41753) Add tests for ArrayZip to check the result size and nullability.

2022-12-28 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-41753: - Summary: Add tests for ArrayZip to check the result size and nullability. Key: SPARK-41753 URL: https://issues.apache.org/jira/browse/SPARK-41753 Project: Spark

[jira] [Created] (SPARK-39419) When the comparator of ArraySort returns null, it should fail.

2022-06-08 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-39419: - Summary: When the comparator of ArraySort returns null, it should fail. Key: SPARK-39419 URL: https://issues.apache.org/jira/browse/SPARK-39419 Project: Spark

[jira] [Created] (SPARK-39293) The accumulator of ArrayAggregate should copy the intermediate result if string, struct, array, or map

2022-05-25 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-39293: - Summary: The accumulator of ArrayAggregate should copy the intermediate result if string, struct, array, or map Key: SPARK-39293 URL:

[jira] [Resolved] (SPARK-39048) Refactor `GroupBy._reduce_for_stat_function` on accepted data types

2022-04-29 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-39048. --- Assignee: Xinrong Meng Resolution: Fixed Issue resolved by pull request 36382

[jira] [Created] (SPARK-38882) The usage logger attachment logic should handle static methods properly.

2022-04-12 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-38882: - Summary: The usage logger attachment logic should handle static methods properly. Key: SPARK-38882 URL: https://issues.apache.org/jira/browse/SPARK-38882 Project:

[jira] [Created] (SPARK-38628) Complete the copy method in subclasses of InternalRow, ArrayData, and MapData to safely copy their instances.

2022-03-22 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-38628: - Summary: Complete the copy method in subclasses of InternalRow, ArrayData, and MapData to safely copy their instances. Key: SPARK-38628 URL:

[jira] [Resolved] (SPARK-38484) Move usage logging instrumentation util functions from pandas module to pyspark.util module

2022-03-15 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-38484. --- Fix Version/s: 3.3.0 Assignee: Yihong He Resolution: Fixed Issue resolved

[jira] [Resolved] (SPARK-37491) Fix Series.asof when values of the series is not sorted

2022-03-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-37491. --- Assignee: pralabhkumar Resolution: Fixed Issue resolved by pull request 35191

[jira] [Resolved] (SPARK-38387) Support `na_action` and Series input correspondence in `Series.map`

2022-03-09 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-38387. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Created] (SPARK-37903) Replace string_typehints with get_type_hints.

2022-01-13 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37903: - Summary: Replace string_typehints with get_type_hints. Key: SPARK-37903 URL: https://issues.apache.org/jira/browse/SPARK-37903 Project: Spark Issue Type:

[jira] [Created] (SPARK-37885) Allow pandas_udf to take type annotations with future annotations enabled

2022-01-12 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37885: - Summary: Allow pandas_udf to take type annotations with future annotations enabled Key: SPARK-37885 URL: https://issues.apache.org/jira/browse/SPARK-37885 Project:

[jira] [Created] (SPARK-37782) Make DataFrame.transform take the parameters for the function.

2021-12-29 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37782: - Summary: Make DataFrame.transform take the parameters for the function. Key: SPARK-37782 URL: https://issues.apache.org/jira/browse/SPARK-37782 Project: Spark

[jira] [Resolved] (SPARK-37678) Incorrect annotations in SeriesGroupBy._cleanup_and_return

2021-12-20 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-37678. --- Fix Version/s: 3.2.1 3.3.0 Assignee: Maciej Szymkiewicz

[jira] [Commented] (SPARK-37678) Incorrect annotations in SeriesGroupBy._cleanup_and_return

2021-12-17 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461681#comment-17461681 ] Takuya Ueshin commented on SPARK-37678: --- Yes! > Incorrect annotations in

[jira] [Comment Edited] (SPARK-37678) Incorrect annotations in SeriesGroupBy._cleanup_and_return

2021-12-17 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461677#comment-17461677 ] Takuya Ueshin edited comment on SPARK-37678 at 12/17/21, 9:30 PM: -- Good

[jira] [Commented] (SPARK-37678) Incorrect annotations in SeriesGroupBy._cleanup_and_return

2021-12-17 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461677#comment-17461677 ] Takuya Ueshin commented on SPARK-37678: --- Good catch! It must be {{{}_cleanup_and_return(self,

[jira] [Commented] (SPARK-37669) Remove unnecessary usages of OrderedDict

2021-12-16 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461057#comment-17461057 ] Takuya Ueshin commented on SPARK-37669: --- I'm working on this. > Remove unnecessary usages of

[jira] [Created] (SPARK-37669) Remove unnecessary usages of OrderedDict

2021-12-16 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37669: - Summary: Remove unnecessary usages of OrderedDict Key: SPARK-37669 URL: https://issues.apache.org/jira/browse/SPARK-37669 Project: Spark Issue Type:

[jira] [Created] (SPARK-37514) Remove workarounds due to older pandas

2021-12-01 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37514: - Summary: Remove workarounds due to older pandas Key: SPARK-37514 URL: https://issues.apache.org/jira/browse/SPARK-37514 Project: Spark Issue Type:

[jira] [Created] (SPARK-37443) Provide a profiler for Python/Pandas UDFs

2021-11-22 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37443: - Summary: Provide a profiler for Python/Pandas UDFs Key: SPARK-37443 URL: https://issues.apache.org/jira/browse/SPARK-37443 Project: Spark Issue Type:

[jira] [Created] (SPARK-37374) StatCounter should use mergeStats when merging with self.

2021-11-18 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37374: - Summary: StatCounter should use mergeStats when merging with self. Key: SPARK-37374 URL: https://issues.apache.org/jira/browse/SPARK-37374 Project: Spark

[jira] [Resolved] (SPARK-37298) Use unique exprId in RewriteAsOfJoin

2021-11-12 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-37298. --- Fix Version/s: 3.3.0 Assignee: Allison Wang Resolution: Fixed Issue

[jira] [Created] (SPARK-37296) Add missing type hints in python/pyspark/util.py

2021-11-11 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37296: - Summary: Add missing type hints in python/pyspark/util.py Key: SPARK-37296 URL: https://issues.apache.org/jira/browse/SPARK-37296 Project: Spark Issue

[jira] [Updated] (SPARK-36845) Inline type hint files for files in python/pyspark/sql

2021-11-11 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-36845: -- Summary: Inline type hint files for files in python/pyspark/sql (was: Inline type hint

[jira] [Commented] (SPARK-36845) Inline type hint files

2021-10-21 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432684#comment-17432684 ] Takuya Ueshin commented on SPARK-36845: --- Hi [~dchvn], shall we file separate umbrella tickets for

[jira] [Created] (SPARK-37079) Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly

2021-10-20 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37079: - Summary: Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly Key: SPARK-37079 URL: https://issues.apache.org/jira/browse/SPARK-37079 Project:

[jira] [Resolved] (SPARK-37048) Clean up inlining type hints under SQL module

2021-10-20 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-37048. --- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Issue

[jira] [Resolved] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py

2021-10-18 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36945. --- Fix Version/s: 3.3.0 Assignee: dch nguyen Resolution: Fixed Issue resolved

[jira] [Commented] (SPARK-37048) Clean up inlining type hints under SQL module

2021-10-18 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430245#comment-17430245 ] Takuya Ueshin commented on SPARK-37048: --- I'm working on this. > Clean up inlining type hints

[jira] [Created] (SPARK-37048) Clean up inlining type hints under SQL module

2021-10-18 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37048: - Summary: Clean up inlining type hints under SQL module Key: SPARK-37048 URL: https://issues.apache.org/jira/browse/SPARK-37048 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-36886) Inline type hints for python/pyspark/sql/context.py

2021-10-18 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36886. --- Fix Version/s: 3.3.0 Assignee: dch nguyen Resolution: Fixed Issue resolved

[jira] [Resolved] (SPARK-36910) Inline type hints for python/pyspark/sql/types.py

2021-10-15 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36910. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Resolved] (SPARK-36991) Inline type hints for spark/python/pyspark/sql/streaming.py

2021-10-15 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36991. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Updated] (SPARK-37011) Upgrade flake8 to 3.9.0 or above in Jenkins

2021-10-14 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-37011: -- Description: In flake8 < 3.9.0, F401 error occurs for imports when the imported identities

<    1   2   3   4   5   6   7   8   9   >