[jira] [Updated] (SPARK-41857) Enable test_between_function, test_datetime_functions, test_expr, test_math_functions, test_window_functions_cumulative_sum, test_corr, test_cov, test_crosstab, test_app

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41857: -- Summary: Enable test_between_function, test_datetime_functions, test_expr,

[jira] [Assigned] (SPARK-41857) Enable test_between_function, test_datetime_functions, test_expr, test_function_parity, test_math_functions, test_window_functions_cumulative_sum, test_corr, test_cov,

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41857: Assignee: Apache Spark (was: Hyukjin Kwon) > Enable test_between_function,

[jira] [Commented] (SPARK-41857) Enable test_between_function, test_datetime_functions, test_expr, test_function_parity, test_math_functions, test_window_functions_cumulative_sum, test_corr, test_cov,

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653829#comment-17653829 ] Apache Spark commented on SPARK-41857: -- User 'techaddict' has created a pull request for this

[jira] [Assigned] (SPARK-41857) Enable test_between_function, test_datetime_functions, test_expr, test_function_parity, test_math_functions, test_window_functions_cumulative_sum, test_corr, test_cov,

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41857: Assignee: Hyukjin Kwon (was: Apache Spark) > Enable test_between_function,

[jira] [Updated] (SPARK-41857) Enable test_between_function, test_datetime_functions, test_expr, test_function_parity, test_math_functions, test_window_functions_cumulative_sum, test_corr, test_cov, t

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41857: -- Summary: Enable test_between_function, test_datetime_functions, test_expr,

[jira] [Created] (SPARK-41857) Enable 10 tests that pass

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41857: - Summary: Enable 10 tests that pass Key: SPARK-41857 URL: https://issues.apache.org/jira/browse/SPARK-41857 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-41856) Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653819#comment-17653819 ] Apache Spark commented on SPARK-41856: -- User 'techaddict' has created a pull request for this

[jira] [Assigned] (SPARK-41856) Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41856: Assignee: Apache Spark (was: Hyukjin Kwon) > Enable test_create_nan_decimal_dataframe,

[jira] [Commented] (SPARK-41856) Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653818#comment-17653818 ] Apache Spark commented on SPARK-41856: -- User 'techaddict' has created a pull request for this

[jira] [Assigned] (SPARK-41856) Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41856: Assignee: Hyukjin Kwon (was: Apache Spark) > Enable test_create_nan_decimal_dataframe,

[jira] [Updated] (SPARK-41856) Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41856: -- Description: 5 tests pass now. Should enable them. (was: These tests pass now. Should enable

[jira] [Created] (SPARK-41856) Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41856: - Summary: Enable test_create_nan_decimal_dataframe, test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found Key: SPARK-41856

[jira] [Commented] (SPARK-41677) Protobuf serializer for StreamingQueryProgressWrapper

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653802#comment-17653802 ] Apache Spark commented on SPARK-41677: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-41677) Protobuf serializer for StreamingQueryProgressWrapper

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41677: Assignee: (was: Apache Spark) > Protobuf serializer for

[jira] [Assigned] (SPARK-41677) Protobuf serializer for StreamingQueryProgressWrapper

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41677: Assignee: Apache Spark > Protobuf serializer for StreamingQueryProgressWrapper >

[jira] [Commented] (SPARK-41677) Protobuf serializer for StreamingQueryProgressWrapper

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653803#comment-17653803 ] Apache Spark commented on SPARK-41677: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-41423) Protobuf serializer for StageDataWrapper

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653798#comment-17653798 ] Apache Spark commented on SPARK-41423: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-40263) Use interruptible lock instead of synchronized in TransportClientFactory.createClient()

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653795#comment-17653795 ] Apache Spark commented on SPARK-40263: -- User 'techaddict' has created a pull request for this

[jira] [Assigned] (SPARK-40263) Use interruptible lock instead of synchronized in TransportClientFactory.createClient()

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40263: Assignee: (was: Apache Spark) > Use interruptible lock instead of synchronized in >

[jira] [Assigned] (SPARK-40263) Use interruptible lock instead of synchronized in TransportClientFactory.createClient()

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40263: Assignee: Apache Spark > Use interruptible lock instead of synchronized in >

[jira] [Commented] (SPARK-40263) Use interruptible lock instead of synchronized in TransportClientFactory.createClient()

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653790#comment-17653790 ] Apache Spark commented on SPARK-40263: -- User 'techaddict' has created a pull request for this

[jira] [Commented] (SPARK-41656) Enable doctests in pyspark.sql.connect.dataframe

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653787#comment-17653787 ] Apache Spark commented on SPARK-41656: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Resolved] (SPARK-41839) Implement SparkSession.sparkContext

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41839. -- Resolution: Invalid I am resolving this because Spark Connect is not designed to support

[jira] [Commented] (SPARK-41656) Enable doctests in pyspark.sql.connect.dataframe

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653786#comment-17653786 ] Apache Spark commented on SPARK-41656: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Resolved] (SPARK-41819) Implement Dataframe.rdd getNumPartitions

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41819. -- Resolution: Invalid I am resolving this because Spark Connect is not designed to support

[jira] [Commented] (SPARK-41658) Enable doctests in pyspark.sql.connect.functions

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653784#comment-17653784 ] Apache Spark commented on SPARK-41658: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-41497) Accumulator undercounting in the case of retry task with rdd cache

2023-01-02 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653781#comment-17653781 ] wuyi commented on SPARK-41497: -- > If I am not wrong, SQL makes very heavy use of accumulators, and so most

[jira] [Resolved] (SPARK-41826) Implement Dataframe.readStream

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41826. -- Resolution: Duplicate > Implement Dataframe.readStream > -- > >

[jira] [Commented] (SPARK-41234) High-order function: array_insert

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653778#comment-17653778 ] Ruifeng Zheng commented on SPARK-41234: --- {code:sql}

[jira] [Assigned] (SPARK-41311) Rewrite test RENAME_SRC_PATH_NOT_FOUND to trigger the error from user space

2023-01-02 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-41311: Assignee: Immanuel Buder > Rewrite test RENAME_SRC_PATH_NOT_FOUND to trigger the error from user

[jira] [Resolved] (SPARK-41311) Rewrite test RENAME_SRC_PATH_NOT_FOUND to trigger the error from user space

2023-01-02 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-41311. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39348

[jira] [Assigned] (SPARK-41661) Support for Python UDFs

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41661: Assignee: Xinrong Meng > Support for Python UDFs > --- > >

[jira] [Commented] (SPARK-41651) Test parity: pyspark.sql.tests.test_dataframe

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653775#comment-17653775 ] Hyukjin Kwon commented on SPARK-41651: -- cc [~techaddict] in case you're interested in this. We can

[jira] [Commented] (SPARK-41652) Test parity: pyspark.sql.tests.test_functions

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653776#comment-17653776 ] Hyukjin Kwon commented on SPARK-41652: -- cc [~techaddict] in case you're interested in this. We can

[jira] [Commented] (SPARK-41658) Enable doctests in pyspark.sql.connect.functions

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653774#comment-17653774 ] Hyukjin Kwon commented on SPARK-41658: -- Fixed in https://github.com/apache/spark/pull/39347 >

[jira] [Resolved] (SPARK-41658) Enable doctests in pyspark.sql.connect.functions

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41658. -- Resolution: Fixed > Enable doctests in pyspark.sql.connect.functions >

[jira] [Resolved] (SPARK-41653) Test parity: enable doctests in Spark Connect

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41653. -- Resolution: Done > Test parity: enable doctests in Spark Connect >

[jira] [Assigned] (SPARK-41658) Enable doctests in pyspark.sql.connect.functions

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41658: Assignee: Sandeep Singh > Enable doctests in pyspark.sql.connect.functions >

[jira] [Assigned] (SPARK-41807) Remove non-existent error class: UNSUPPORTED_FEATURE.DISTRIBUTE_BY

2023-01-02 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-41807: Assignee: BingKun Pan > Remove non-existent error class: UNSUPPORTED_FEATURE.DISTRIBUTE_BY >

[jira] [Resolved] (SPARK-41807) Remove non-existent error class: UNSUPPORTED_FEATURE.DISTRIBUTE_BY

2023-01-02 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-41807. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39335

[jira] [Assigned] (SPARK-41841) Support PyPI packaging without JVM

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41841: Assignee: Apache Spark > Support PyPI packaging without JVM >

[jira] [Commented] (SPARK-41841) Support PyPI packaging without JVM

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653770#comment-17653770 ] Apache Spark commented on SPARK-41841: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-41841) Support PyPI packaging without JVM

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41841: Assignee: (was: Apache Spark) > Support PyPI packaging without JVM >

[jira] [Resolved] (SPARK-41854) Automatic reformat/check python/setup.py

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41854. -- Fix Version/s: 3.4.0 Assignee: Hyukjin Kwon Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-41855) `createDataFrame` doesn't handle None/NaN properly

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-41855: -- Description: {code:python} data = [Row(id=1, value=float("NaN")), Row(id=2,

[jira] [Updated] (SPARK-41855) `createDataFrame` doesn't handle None/NaN properly

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-41855: -- Description: {code:python} data = [Row(id=1, value=float("NaN")), Row(id=2,

[jira] [Updated] (SPARK-41855) `createDataFrame` doesn't handle None/NaN properly

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-41855: -- Summary: `createDataFrame` doesn't handle None/NaN properly (was: `createDataFrame` doesn't

[jira] [Created] (SPARK-41855) `createDataFrame` doesn't handle None properly

2023-01-02 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-41855: - Summary: `createDataFrame` doesn't handle None properly Key: SPARK-41855 URL: https://issues.apache.org/jira/browse/SPARK-41855 Project: Spark Issue Type:

[jira] [Commented] (SPARK-41854) Automatic reformat/check python/setup.py

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653756#comment-17653756 ] Apache Spark commented on SPARK-41854: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-41854) Automatic reformat/check python/setup.py

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41854: Assignee: Apache Spark > Automatic reformat/check python/setup.py >

[jira] [Assigned] (SPARK-41854) Automatic reformat/check python/setup.py

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41854: Assignee: (was: Apache Spark) > Automatic reformat/check python/setup.py >

[jira] [Created] (SPARK-41854) Automatic reformat/check python/setup.py

2023-01-02 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-41854: Summary: Automatic reformat/check python/setup.py Key: SPARK-41854 URL: https://issues.apache.org/jira/browse/SPARK-41854 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-39995) PySpark installation doesn't support Scala 2.13 binaries

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653753#comment-17653753 ] Hyukjin Kwon commented on SPARK-39995: -- I think i will be able to pick this up before Spark 3.4. >

[jira] [Commented] (SPARK-39995) PySpark installation doesn't support Scala 2.13 binaries

2023-01-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653752#comment-17653752 ] Hyukjin Kwon commented on SPARK-39995: -- For: {quote} Also, env vars (e.g. PYSPARK_HADOOP_VERSION)

[jira] [Commented] (SPARK-41852) Fix `pmod` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653750#comment-17653750 ] Sandeep Singh commented on SPARK-41852: --- [~podongfeng] these are from the doctests {code:java} >>>

[jira] [Commented] (SPARK-41851) Fix `nanvl` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653751#comment-17653751 ] Sandeep Singh commented on SPARK-41851: --- [~podongfeng]  {code:java} >>> df =

[jira] [Commented] (SPARK-41815) Column.isNull returns nan instead of None

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653748#comment-17653748 ] Ruifeng Zheng commented on SPARK-41815: --- similar to the issue in `createDataFrame`

[jira] [Comment Edited] (SPARK-41814) Column.eqNullSafe fails on NaN comparison

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653736#comment-17653736 ] Ruifeng Zheng edited comment on SPARK-41814 at 1/3/23 3:12 AM: --- this issue

[jira] [Commented] (SPARK-41851) Fix `nanvl` function

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653746#comment-17653746 ] Ruifeng Zheng commented on SPARK-41851: --- could you please also provide the code to create the

[jira] [Comment Edited] (SPARK-41814) Column.eqNullSafe fails on NaN comparison

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653736#comment-17653736 ] Ruifeng Zheng edited comment on SPARK-41814 at 1/3/23 3:06 AM: --- this issue

[jira] [Commented] (SPARK-41852) Fix `pmod` function

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653743#comment-17653743 ] Ruifeng Zheng commented on SPARK-41852: --- could you please also provide the code to create the

[jira] [Commented] (SPARK-41853) Use Map in place of SortedMap for ErrorClassesJsonReader

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653741#comment-17653741 ] Apache Spark commented on SPARK-41853: -- User 'tedyu' has created a pull request for this issue:

[jira] [Updated] (SPARK-41847) DataFrame mapfield,structlist invalid type

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41847: -- Description: {code:java} File

[jira] [Commented] (SPARK-41853) Use Map in place of SortedMap for ErrorClassesJsonReader

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653740#comment-17653740 ] Apache Spark commented on SPARK-41853: -- User 'tedyu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-41853) Use Map in place of SortedMap for ErrorClassesJsonReader

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41853: Assignee: Apache Spark > Use Map in place of SortedMap for ErrorClassesJsonReader >

[jira] [Assigned] (SPARK-41853) Use Map in place of SortedMap for ErrorClassesJsonReader

2023-01-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41853: Assignee: (was: Apache Spark) > Use Map in place of SortedMap for

[jira] [Created] (SPARK-41853) Use Map in place of SortedMap for ErrorClassesJsonReader

2023-01-02 Thread Ted Yu (Jira)
Ted Yu created SPARK-41853: -- Summary: Use Map in place of SortedMap for ErrorClassesJsonReader Key: SPARK-41853 URL: https://issues.apache.org/jira/browse/SPARK-41853 Project: Spark Issue Type:

[jira] [Created] (SPARK-41852) Fix `pmod` function

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41852: - Summary: Fix `pmod` function Key: SPARK-41852 URL: https://issues.apache.org/jira/browse/SPARK-41852 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-41852) Fix `pmod` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41852: -- Description: {code:java} File

[jira] [Updated] (SPARK-41848) Tasks are over-scheduled with TaskResourceProfile

2023-01-02 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-41848: - Priority: Blocker (was: Major) > Tasks are over-scheduled with TaskResourceProfile >

[jira] [Commented] (SPARK-41848) Tasks are over-scheduled with TaskResourceProfile

2023-01-02 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653739#comment-17653739 ] wuyi commented on SPARK-41848: -- cc [~ivoson]  > Tasks are over-scheduled with TaskResourceProfile >

[jira] [Created] (SPARK-41851) Fix `nanvl` function

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41851: - Summary: Fix `nanvl` function Key: SPARK-41851 URL: https://issues.apache.org/jira/browse/SPARK-41851 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-41851) Fix `nanvl` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41851: -- Description: {code:java} File

[jira] [Updated] (SPARK-41847) DataFrame mapfield,structlist invalid type

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41847: -- Description: {code:java} File

[jira] [Commented] (SPARK-41850) Fix `isnan` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653738#comment-17653738 ] Sandeep Singh commented on SPARK-41850: --- This should be moved under SPARK-41283 > Fix `isnan`

[jira] [Updated] (SPARK-41850) Fix `isnan` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41850: -- Description: {code:java} File

[jira] [Updated] (SPARK-41850) Fix `isnan` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41850: -- Summary: Fix `isnan` function (was: Fix DataFrameReader.isnan) > Fix `isnan` function >

[jira] [Created] (SPARK-41850) Fix DataFrameReader.isnan

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41850: - Summary: Fix DataFrameReader.isnan Key: SPARK-41850 URL: https://issues.apache.org/jira/browse/SPARK-41850 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-41814) Column.eqNullSafe fails on NaN comparison

2023-01-02 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653736#comment-17653736 ] Ruifeng Zheng commented on SPARK-41814: --- this issue is due to: 1, the conversion from rows to

[jira] [Created] (SPARK-41849) Implement DataFrameReader.text

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41849: - Summary: Implement DataFrameReader.text Key: SPARK-41849 URL: https://issues.apache.org/jira/browse/SPARK-41849 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-41849) Implement DataFrameReader.text

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41849: -- Description: {code:java} File

[jira] [Created] (SPARK-41848) Tasks are over-scheduled with TaskResourceProfile

2023-01-02 Thread wuyi (Jira)
wuyi created SPARK-41848: Summary: Tasks are over-scheduled with TaskResourceProfile Key: SPARK-41848 URL: https://issues.apache.org/jira/browse/SPARK-41848 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-41847) DataFrame mapfield,structlist invalid type

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41847: -- Description: {code:java} File

[jira] [Updated] (SPARK-41847) DataFrame mapfield,structlist invalid type

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41847: -- Summary: DataFrame mapfield,structlist invalid type (was: DataFrame mapfield invalid type)

[jira] [Updated] (SPARK-41847) DataFrame mapfield invalid type

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41847: -- Description: {code:java} File

[jira] [Created] (SPARK-41847) DataFrame mapfield invalid type

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41847: - Summary: DataFrame mapfield invalid type Key: SPARK-41847 URL: https://issues.apache.org/jira/browse/SPARK-41847 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-41846) DataFrame windowspec functions : unresolved columns

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41846: -- Summary: DataFrame windowspec functions : unresolved columns (was: DataFrame aggregation

[jira] [Updated] (SPARK-41846) DataFrame aggregation functions : unresolved columns

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41846: -- Description: {code:java} File

[jira] [Updated] (SPARK-39853) Support stage level schedule for standalone cluster when dynamic allocation is disabled

2023-01-02 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-39853: - Fix Version/s: 3.4.0 > Support stage level schedule for standalone cluster when dynamic allocation > is

[jira] [Updated] (SPARK-41846) DataFrame aggregation functions : unresolved columns

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41846: -- Description: {code:java} File

[jira] [Created] (SPARK-41846) DataFrame aggregation functions : unresolved columns

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41846: - Summary: DataFrame aggregation functions : unresolved columns Key: SPARK-41846 URL: https://issues.apache.org/jira/browse/SPARK-41846 Project: Spark Issue

[jira] [Commented] (SPARK-37677) spark on k8s, when the user want to push python3.6.6.zip to the pod , but no permission to execute

2023-01-02 Thread jingxiong zhong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653733#comment-17653733 ] jingxiong zhong commented on SPARK-37677: - At present, I have repaired Hadoop version 3.3.5, but

[jira] [Resolved] (SPARK-37521) insert overwrite table but the partition information stored in Metastore was not changed

2023-01-02 Thread jingxiong zhong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jingxiong zhong resolved SPARK-37521. - Resolution: Won't Fix > insert overwrite table but the partition information stored in

[jira] [Resolved] (SPARK-41823) DataFrame.join creating ambiguous column names

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh resolved SPARK-41823. --- Resolution: Duplicate > DataFrame.join creating ambiguous column names >

[jira] [Updated] (SPARK-41845) Fix `count(expr("*"))` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41845: -- Description: {code:java} File

[jira] [Created] (SPARK-41845) Fix `count(expr("*"))` function

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41845: - Summary: Fix `count(expr("*"))` function Key: SPARK-41845 URL: https://issues.apache.org/jira/browse/SPARK-41845 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-41844) Implement `intX2` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh resolved SPARK-41844. --- Resolution: Invalid > Implement `intX2` function > -- > >

[jira] [Created] (SPARK-41844) Implement `intX2` function

2023-01-02 Thread Sandeep Singh (Jira)
Sandeep Singh created SPARK-41844: - Summary: Implement `intX2` function Key: SPARK-41844 URL: https://issues.apache.org/jira/browse/SPARK-41844 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-41844) Implement `intX2` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41844: -- Description: {code:java} File

[jira] [Updated] (SPARK-41835) Implement `transform_keys` function

2023-01-02 Thread Sandeep Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Singh updated SPARK-41835: -- Description: {code:java} File

  1   2   3   >