[
https://issues.apache.org/jira/browse/SPARK-37697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626850#comment-17626850
]
Xinrong Meng commented on SPARK-37697:
--------------------------------------
Hi, we have NumPy input support
https://issues.apache.org/jira/browse/SPARK-39405 in Spark 3.4.0.
> Make it easier to convert numpy arrays to Spark Dataframes
> ----------------------------------------------------------
>
> Key: SPARK-37697
> URL: https://issues.apache.org/jira/browse/SPARK-37697
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 3.1.2
> Reporter: Douglas Moore
> Priority: Major
>
> Make it easier to convert numpy arrays to dataframes.
> Often we receive errors:
>
> {code:java}
> df = spark.createDataFrame(numpy.arange(10))
> Can not infer schema for type: <class 'numpy.int64'>
> {code}
>
> OR
> {code:java}
> df = spark.createDataFrame(numpy.arange(10.))
> Can not infer schema for type: <class 'numpy.float64'>
> {code}
>
> Today (Spark 3.x) we have to:
> {code:java}
> spark.createDataFrame(pd.DataFrame(numpy.arange(10.))) {code}
> Make this easier with a direct conversion from Numpy arrays to Spark
> Dataframes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
