Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

Hi Everyone, I'm an Arrow contributor mostly on the C++ side of things, but I'll try to give a brief update of where I believe the project currently is (the views are my own, but hopefully are fairly accurate :). I think in the long run the diagram mentioned by Jim, is were we would like Arrow to

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

Don't know much about Spark + Arrow efforts myself; just wanted to share the reference. On Fri, Aug 5, 2016 at 6:53 PM Jim Pivarski wrote: > On Fri, Aug 5, 2016 at 5:14 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Relevant jira:

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

On Fri, Aug 5, 2016 at 5:14 PM, Nicholas Chammas wrote: > Relevant jira: https://issues.apache.org/jira/browse/SPARK-13534 > Thank you. This ticket describes output from Spark to Arrow for flat (non-nested) tables. Are there no plans to input from Arrow to Spark for

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

t;>>> >>>> I recently learned about Apache Arrow, a data layer that Spark >>>> currently or >>>> will someday share with Pandas, Impala, etc. Suppose that I can fill a >>>> buffer (such as a direct ByteBuffer) with Arrow-formatted data, i

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

r even zero-copy--- way to use that in Spark? Is that an API >>> that >>> could be developed? >>> >>> I'll be at the KDD Spark 2.0 tutorial on August 15. Is that a good place >>> to >>> ask this question? >>> >>> Thanks, >>>

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

ata, is there >> an >> easy--- or even zero-copy--- way to use that in Spark? Is that an API that >> could be developed? >> >> I'll be at the KDD Spark 2.0 tutorial on August 15. Is that a good place >> to >> ask this question? >> >> Thanks, >

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

s that a good place >> to >> ask this question? >> >> Thanks, >> -- Jim >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Arrow-data-in-bu

Re: Apache Arrow data in buffer to RDD/DataFrame/Dataset?

there an > easy--- or even zero-copy--- way to use that in Spark? Is that an API that > could be developed? > > I'll be at the KDD Spark 2.0 tutorial on August 15. Is that a good place to > ask this question? > > Thanks, > -- Jim > > > > > -- > View this message in c

Apache Arrow data in buffer to RDD/DataFrame/Dataset?

on August 15. Is that a good place to ask this question? Thanks, -- Jim -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Arrow-data-in-buffer-to-RDD-DataFrame-Dataset-tp18563.html Sent from the Apache Spark Developers List mailing list archive a