Xinrong Meng created SPARK-41150:
Summary: Document PySpark memory profiler
Key: SPARK-41150
URL: https://issues.apache.org/jira/browse/SPARK-41150
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39405:
-
Description:
NumPy is the fundamental package for scientific computing with Python. It is
very
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39405:
-
Summary: NumPy input support in PySpark (was: NumPy input support in
PySpark SQL)
> NumPy
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39405:
-
Summary: NumPy Input Support in PySpark (was: NumPy input support in
PySpark)
> NumPy Input
[
https://issues.apache.org/jira/browse/SPARK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-41107:
-
Description: PySpark memory profiler depends on
[
https://issues.apache.org/jira/browse/SPARK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-41107:
-
Description:
PySpark memory profiler depends on
[
https://issues.apache.org/jira/browse/SPARK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-41107:
-
Description: PySpark memory profiler depends on
[
https://issues.apache.org/jira/browse/SPARK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-41107:
-
Summary: Install memory-profiler in the CI (was: Install memory-profiler
in CI)
> Install
[
https://issues.apache.org/jira/browse/SPARK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-41107:
-
Summary: Install memory-profiler in CI (was: Install the Memory Profiler
in CI)
> Install
[
https://issues.apache.org/jira/browse/SPARK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-41107:
-
Component/s: Tests
> Install the Memory Profiler in CI
> -
>
>
Xinrong Meng created SPARK-41107:
Summary: Install the Memory Profiler in CI
Key: SPARK-41107
URL: https://issues.apache.org/jira/browse/SPARK-41107
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Description:
The ticket proposes to implement PySpark memory profiling on executors. See
more
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Description:
The ticket proposes to implement PySpark memory profiling on executors. See
more
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Description:
The ticket proposes to implement PySpark memory profiling on executors. See
more
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Description:
The ticket proposes to implement a PySpark memory profiling on executors. See
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Description:
Profiling is critical to performance engineering. Memory consumption is a key
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630120#comment-17630120
]
Xinrong Meng commented on SPARK-40281:
--
Thanks [~alfiewdavidson] for the feedback!
I am currently
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627984#comment-17627984
]
Xinrong Meng edited comment on SPARK-39405 at 11/2/22 9:10 PM:
---
Hi
[
https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627985#comment-17627985
]
Xinrong Meng commented on SPARK-37697:
--
The commit is in.
> Make it easier to convert numpy arrays
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627983#comment-17627983
]
Xinrong Meng commented on SPARK-40990:
--
Hi [~douglas.mo...@databricks.com] Any size of the 2d
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627984#comment-17627984
]
Xinrong Meng commented on SPARK-39405:
--
Hi [~douglas.mo...@databricks.com] the commit is in.
>
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627359#comment-17627359
]
Xinrong Meng commented on SPARK-39405:
--
Thanks [~douglas.mo...@databricks.com] , your queries
[
https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627357#comment-17627357
]
Xinrong Meng commented on SPARK-37697:
--
Thanks [~douglas.mo...@databricks.com] , your queries
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40990:
-
Summary: DataFrame creation from 2d NumPy array with arbitrary columns
(was: Complete support
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40990:
-
Description:
Currently, DataFrame creation from 2d ndarray works only with 2 columns. We
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40990:
-
Summary: Complete support for DataFrame creation from 2d NumPy array (was:
Support DataFrame
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40990:
-
Description:
Currently, DataFrame creation from ndarray works only with <= 2 columns. We
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40990:
-
Description:
Currently, DataFrame creation from ndarray works only with <= 2 columns. We
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40990:
-
Description:
Currently, DataFrame creation from ndarray works only with <= 2 columns. We
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627337#comment-17627337
]
Xinrong Meng commented on SPARK-40990:
--
I am working on that.
> Support DataFrame creation from
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17627336#comment-17627336
]
Xinrong Meng commented on SPARK-39405:
--
Hi [~douglas.mo...@databricks.com] thanks for the bug
[
https://issues.apache.org/jira/browse/SPARK-40990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-40990:
Assignee: (was: Xinrong Meng)
> Support DataFrame creation from ndarray with >2
Xinrong Meng created SPARK-40990:
Summary: Support DataFrame creation from ndarray with >2 columns
Key: SPARK-40990
URL: https://issues.apache.org/jira/browse/SPARK-40990
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626850#comment-17626850
]
Xinrong Meng commented on SPARK-37697:
--
Hi, we have NumPy input support
[
https://issues.apache.org/jira/browse/SPARK-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626847#comment-17626847
]
Xinrong Meng commented on SPARK-6857:
-
Hi, we have NumPy input support
[
https://issues.apache.org/jira/browse/SPARK-31776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626844#comment-17626844
]
Xinrong Meng commented on SPARK-31776:
--
`lit` supports Python list and NumPy arrays in
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39405:
-
Summary: NumPy input support in PySpark SQL (was: NumPy input support in
PySpark)
> NumPy
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39405:
-
Summary: NumPy input support in PySpark (was: NumPy input support in
PySpark SQL)
> NumPy
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39405:
-
Summary: NumPy input support in PySpark SQL (was: NumPy support in SQL)
> NumPy input support
[
https://issues.apache.org/jira/browse/SPARK-39199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-39199.
--
Resolution: Resolved
> Implement pandas API missing parameters
>
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Description:
Profiling is critical to performance engineering. Memory consumption is a key
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Issue Type: New Feature (was: Umbrella)
> Memory Profiler on Executors
>
[
https://issues.apache.org/jira/browse/SPARK-40598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17613843#comment-17613843
]
Xinrong Meng commented on SPARK-40598:
--
Resolved by [https://github.com/apache/spark/pull/38033].
[
https://issues.apache.org/jira/browse/SPARK-40598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-40598:
Assignee: Haejoon Lee
> Fix plotting features work properly with pandas 1.5.0.
>
[
https://issues.apache.org/jira/browse/SPARK-40598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-40598.
--
Resolution: Resolved
> Fix plotting features work properly with pandas 1.5.0.
>
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40281:
-
Description:
Profiling is critical to performance engineering. Memory consumption is a key
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reopened SPARK-39494:
--
> Support `createDataFrame` from a list of scalars when schema is not provided
>
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-39494.
--
Resolution: Won't Do
> Support `createDataFrame` from a list of scalars when schema is not
[
https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-40084.
--
Resolution: Resolved
> Upgrade Py4J from 0.10.9.5 to 0.10.9.7
>
[
https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606813#comment-17606813
]
Xinrong Meng commented on SPARK-40084:
--
Resolved by https://github.com/apache/spark/pull/37523.
>
[
https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-40084:
Assignee: BingKun Pan
> Upgrade Py4J from 0.10.9.5 to 0.10.9.7
>
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-39405.
--
Resolution: Resolved
> NumPy support in SQL
>
>
> Key:
[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-39405:
Assignee: Xinrong Meng
> NumPy support in SQL
>
>
>
[
https://issues.apache.org/jira/browse/SPARK-39745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-39745.
--
Resolution: Won't Do
> Accept a list that contains NumPy scalars in `createDataFrame`
>
[
https://issues.apache.org/jira/browse/SPARK-40196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40196:
-
Description:
Per [https://github.com/apache/spark/pull/37560#discussion_r952882996,]
function
[
https://issues.apache.org/jira/browse/SPARK-40196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40196:
-
Summary: Consolidate `lit` function with NumPy scalar in sql and pandas
module (was:
[
https://issues.apache.org/jira/browse/SPARK-40309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng deleted SPARK-40309:
-
> Introduce sql_conf context manager for pyspark.sql
>
[
https://issues.apache.org/jira/browse/SPARK-40131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-40131.
--
Resolution: Resolved
> Support NumPy ndarray in built-in functions
>
[
https://issues.apache.org/jira/browse/SPARK-40131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603201#comment-17603201
]
Xinrong Meng commented on SPARK-40131:
--
Resolved by [https://github.com/apache/spark/pull/37635.]
[
https://issues.apache.org/jira/browse/SPARK-40131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-40131:
Assignee: Xinrong Meng
> Support NumPy ndarray in built-in functions
>
[
https://issues.apache.org/jira/browse/SPARK-40309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40309:
-
Description:
That would simplify the control of Spark SQL configuration as below
from
[
https://issues.apache.org/jira/browse/SPARK-40309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40309:
-
Description:
[
https://issues.apache.org/jira/browse/SPARK-40309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40309:
-
Description:
[https://github.com/apache/spark/blob/master/python/pyspark/pandas/utils.py#L490]
Xinrong Meng created SPARK-40309:
Summary: Introduce sql_conf context manager for pyspark.sql
Key: SPARK-40309
URL: https://issues.apache.org/jira/browse/SPARK-40309
Project: Spark
Issue
Xinrong Meng created SPARK-40307:
Summary: Optimize (De)Serialization of Python UDF
Key: SPARK-40307
URL: https://issues.apache.org/jira/browse/SPARK-40307
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-40281:
Assignee: (was: Xinrong Meng)
> Memory Profiler on Executors
>
Xinrong Meng created SPARK-40281:
Summary: Memory Profiler on Executors
Key: SPARK-40281
URL: https://issues.apache.org/jira/browse/SPARK-40281
Project: Spark
Issue Type: Umbrella
[
https://issues.apache.org/jira/browse/SPARK-40281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-40281:
Assignee: Xinrong Meng
> Memory Profiler on Executors
>
>
>
[
https://issues.apache.org/jira/browse/SPARK-40131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40131:
-
Description: Support NumPy ndarray in built-in
functions(`pyspark.sql.functions`) by
[
https://issues.apache.org/jira/browse/SPARK-40130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-40130.
--
Assignee: Xinrong Meng
Resolution: Fixed
> Support NumPy scalars in built-in functions
[
https://issues.apache.org/jira/browse/SPARK-39483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-39483:
Assignee: Xinrong Meng
> Construct the schema from `np.dtype` when `createDataFrame`
[
https://issues.apache.org/jira/browse/SPARK-40130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584975#comment-17584975
]
Xinrong Meng commented on SPARK-40130:
--
Resolved by https://github.com/apache/spark/pull/37560
>
[
https://issues.apache.org/jira/browse/SPARK-40196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-40196:
-
Description:
Per [https://github.com/apache/spark/pull/37560#discussion_r952882996,]
function
Xinrong Meng created SPARK-40196:
Summary: Consolidate `lit` function with NumPy input in sql and
pandas module
Key: SPARK-40196
URL: https://issues.apache.org/jira/browse/SPARK-40196
Project: Spark
Xinrong Meng created SPARK-40131:
Summary: Support NumPy ndarray in built-in functions
Key: SPARK-40131
URL: https://issues.apache.org/jira/browse/SPARK-40131
Project: Spark
Issue Type:
Xinrong Meng created SPARK-40130:
Summary: Support NumPy scalars in built-in functions
Key: SPARK-40130
URL: https://issues.apache.org/jira/browse/SPARK-40130
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-40090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-40090.
--
Resolution: Duplicate
> Upgrade to Py4J 0.10.9.7
>
>
>
Xinrong Meng created SPARK-40090:
Summary: Upgrade to Py4J 0.10.9.7
Key: SPARK-40090
URL: https://issues.apache.org/jira/browse/SPARK-40090
Project: Spark
Issue Type: Improvement
Xinrong Meng created SPARK-39986:
Summary: Better example for Co-grouped Map
Key: SPARK-39986
URL: https://issues.apache.org/jira/browse/SPARK-39986
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-39822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39822:
-
Parent: SPARK-39581
Issue Type: Sub-task (was: Bug)
> Provides a good error during
[
https://issues.apache.org/jira/browse/SPARK-39794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568139#comment-17568139
]
Xinrong Meng commented on SPARK-39794:
--
I am working on that.
> Introduce parametric singleton for
[
https://issues.apache.org/jira/browse/SPARK-39794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39794:
-
Description:
As per
Xinrong Meng created SPARK-39794:
Summary: Introduce parametric singleton for DataType
Key: SPARK-39794
URL: https://issues.apache.org/jira/browse/SPARK-39794
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-39761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-39761.
--
Fix Version/s: 3.4.0
Resolution: Fixed
> Add Apache Spark images info in
[
https://issues.apache.org/jira/browse/SPARK-39761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng reassigned SPARK-39761:
Assignee: Yikun Jiang
> Add Apache Spark images info in running-on-kubernetes doc
>
[
https://issues.apache.org/jira/browse/SPARK-39761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17567037#comment-17567037
]
Xinrong Meng commented on SPARK-39761:
--
Fixed in [https://github.com/apache/spark/pull/37174]
>
[
https://issues.apache.org/jira/browse/SPARK-39732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17566928#comment-17566928
]
Xinrong Meng commented on SPARK-39732:
--
Thanks [~itsmeandy] for raising that!
Previously, the
[
https://issues.apache.org/jira/browse/SPARK-39732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17566924#comment-17566924
]
Xinrong Meng commented on SPARK-39732:
--
How about we match pandas behavior/results and utilize
Xinrong Meng created SPARK-39756:
Summary: Better error messages for missing pandas scalars
Key: SPARK-39756
URL: https://issues.apache.org/jira/browse/SPARK-39756
Project: Spark
Issue Type:
Xinrong Meng created SPARK-39745:
Summary: Accept a list that contains NumPy scalars in
`createDataFrame`
Key: SPARK-39745
URL: https://issues.apache.org/jira/browse/SPARK-39745
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39494:
-
Description:
Currently, DataFrame creation from a list of native Python scalars is
unsupported
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39494:
-
Description:
{{>>> spark.createDataFrame([1, 2]).collect()}}
{{Traceback (most recent call
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39494:
-
Description:
{{Currently, DataFrame creation from a list of native Python scalars is
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39494:
-
Description:
{{Currently, DataFrame creation from a list of scalars is unsupported in
PySpark,
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39494:
-
Description:
Currently, DataFrame creation from a list of scalars is unsupported in PySpark,
[
https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39494:
-
Summary: Support `createDataFrame` from a list of scalars when schema is
not provided (was:
[
https://issues.apache.org/jira/browse/SPARK-38953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-38953:
-
Description:
There are common exceptions/errors in PySpark SQL, pandas API on Spark, and
Py4J.
[
https://issues.apache.org/jira/browse/SPARK-39076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng resolved SPARK-39076.
--
Resolution: Done
> Standardize Statistical Functions of pandas API on Spark
>
[
https://issues.apache.org/jira/browse/SPARK-39227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39227:
-
Parent: (was: SPARK-39076)
Issue Type: Improvement (was: Sub-task)
> Reach parity
Xinrong Meng created SPARK-39648:
Summary: Fix type hints of `like`, `rlike`, `ilike` of Column
Key: SPARK-39648
URL: https://issues.apache.org/jira/browse/SPARK-39648
Project: Spark
Issue
401 - 500 of 918 matches
Mail list logo
