Hi Radu, I'm not a spark expert, but I haven't seen any documentation on direct conversion. You might be better off asking the user@spark or dev@spark mailing lists.
Thanks, Micah On Wed, Aug 26, 2020 at 1:46 PM Radu Teodorescu <radukay...@yahoo.com.invalid> wrote: > Hi, > I noticed that arrow is mentioned as an optional intermediary format for > converting between pandas DFs and spark DFs. Is there a way to explicitly > convert an pyarrow Table to a spark DataFrame and the other way around. > Absent that, going pysprak->pandas->pyarrow and back works but it’s > obviously suboptimal. > Thank you > Radu
