Hi Bryan, I came across SPARK-29040 <https://issues.apache.org/jira/browse/SPARK-29040> and I'm very excited that others are looking for such feature as well. It will be tremendously useful if we can implement this feature.
Currently, my workaround is to serialize `pyarrow.Table` to a parquet file, then let Spark to read that parquet file. I avoided using `pd.Dataframe`, same as what Artem mentioned above. Do you think this ticket has a chance to get prioritized? Thank you very much. Best, Shouheng -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
