[
https://issues.apache.org/jira/browse/SPARK-50323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-50323:
---------------------------------
Description:
Enforce schema for createDataFrame from numpy ndarray
Currently, the conversion from ndarray to pa.table doesn’t consider the schema
at all ([for
e.g.|https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py#L649]),
whereas pa.table or pd.DataFrame to pa.table does take the schema into account
([for
e.g.|https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py#L616]).
was:Enforce schema for createDataFrame from numpy ndarray
> Enforce schema for createDataFrame from numpy ndarray
> -----------------------------------------------------
>
> Key: SPARK-50323
> URL: https://issues.apache.org/jira/browse/SPARK-50323
> Project: Spark
> Issue Type: Sub-task
> Components: Connect, PySpark
> Affects Versions: 4.0.0
> Reporter: Xinrong Meng
> Priority: Major
>
> Enforce schema for createDataFrame from numpy ndarray
>
> Currently, the conversion from ndarray to pa.table doesn’t consider the
> schema at all ([for
> e.g.|https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py#L649]),
> whereas pa.table or pd.DataFrame to pa.table does take the schema into
> account ([for
> e.g.|https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/session.py#L616]).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
