Re: Does spark support Apache Arrow
Kwon, Isn't that JIRA is part of integration with Arrow. As far as arrow as in-memory store goes it probably conflicts with spark's own tungsten memory representation, right? Thanks Nir On Thu, May 19, 2016 at 8:03 PM, Hyukjin Kwon wrote: > FYI, there is a JIRA for this, https://issues.apache. > org/jira/browse/SPARK-13534 > > I hope this link is helpful. > > Thanks! > > > 2016-05-20 11:18 GMT+09:00 Sun Rui : > >> 1. I don’t think so >> 2. Arrow is for in-memory columnar execution. While cache is for >> in-memory columnar storage >> >> On May 20, 2016, at 10:16, Todd wrote: >> >> From the official site http://arrow.apache.org/, Apache Arrow is used >> for Columnar In-Memory storage. I have two quick questions: >> 1. Does spark support Apache Arrow? >> 2. When dataframe is cached in memory, the data are saved in columnar >> in-memory style. What is the relationship between this feature and Apache >> Arrow,that is, >> when the data is in Apache Arrow format,does spark still need the effort >> to cache the dataframe in columnar in-memory? >> >> >> > -- [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] <https://twitter.com/Xactly> [image: Facebook] <https://www.facebook.com/XactlyCorp> [image: YouTube] <http://www.youtube.com/xactlycorporation>
Re: Does spark support Apache Arrow
FYI, there is a JIRA for this, https://issues.apache.org/jira/browse/SPARK-13534 I hope this link is helpful. Thanks! 2016-05-20 11:18 GMT+09:00 Sun Rui : > 1. I don’t think so > 2. Arrow is for in-memory columnar execution. While cache is for in-memory > columnar storage > > On May 20, 2016, at 10:16, Todd wrote: > > From the official site http://arrow.apache.org/, Apache Arrow is used for > Columnar In-Memory storage. I have two quick questions: > 1. Does spark support Apache Arrow? > 2. When dataframe is cached in memory, the data are saved in columnar > in-memory style. What is the relationship between this feature and Apache > Arrow,that is, > when the data is in Apache Arrow format,does spark still need the effort > to cache the dataframe in columnar in-memory? > > >
Re: Does spark support Apache Arrow
1. I don’t think so 2. Arrow is for in-memory columnar execution. While cache is for in-memory columnar storage > On May 20, 2016, at 10:16, Todd wrote: > > From the official site http://arrow.apache.org/, Apache Arrow is used for > Columnar In-Memory storage. I have two quick questions: > 1. Does spark support Apache Arrow? > 2. When dataframe is cached in memory, the data are saved in columnar > in-memory style. What is the relationship between this feature and Apache > Arrow,that is, > when the data is in Apache Arrow format,does spark still need the effort to > cache the dataframe in columnar in-memory?
Does spark support Apache Arrow
From the official site http://arrow.apache.org/, Apache Arrow is used for Columnar In-Memory storage. I have two quick questions: 1. Does spark support Apache Arrow? 2. When dataframe is cached in memory, the data are saved in columnar in-memory style. What is the relationship between this feature and Apache Arrow,that is, when the data is in Apache Arrow format,does spark still need the effort to cache the dataframe in columnar in-memory?