Re: Does spark support Apache Arrow

[Nirav Patel]

Kwon,

Isn't that JIRA is part of integration with Arrow.  As far as arrow as
in-memory store goes it probably conflicts with spark's own tungsten memory
representation, right?

Thanks
Nir

On Thu, May 19, 2016 at 8:03 PM, Hyukjin Kwon <gurwls...@gmail.com> wrote:

> FYI, there is a JIRA for this, https://issues.apache.
> org/jira/browse/SPARK-13534
>
> I hope this link is helpful.
>
> Thanks!
>
>
> 2016-05-20 11:18 GMT+09:00 Sun Rui <sunrise_...@163.com>:
>
>> 1. I don’t think so
>> 2. Arrow is for in-memory columnar execution. While cache is for
>> in-memory columnar storage
>>
>> On May 20, 2016, at 10:16, Todd <bit1...@163.com> wrote:
>>
>> From the official site http://arrow.apache.org/, Apache Arrow is used
>> for Columnar In-Memory storage. I have two quick questions:
>> 1. Does spark support Apache Arrow?
>> 2. When dataframe is cached in memory, the data are saved in columnar
>> in-memory style. What is the relationship between this feature and Apache
>> Arrow,that is,
>> when the data is in Apache Arrow format，does spark still need the effort
>> to cache the dataframe in columnar in-memory?
>>
>>
>>
>

-- 


[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>

<https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn] 
<https://www.linkedin.com/company/xactly-corporation>  [image: Twitter] 
<https://twitter.com/Xactly>  [image: Facebook] 
<https://www.facebook.com/XactlyCorp>  [image: YouTube] 
<http://www.youtube.com/xactlycorporation>

Re: Does spark support Apache Arrow

[Hyukjin Kwon]

FYI, there is a JIRA for this,
https://issues.apache.org/jira/browse/SPARK-13534

I hope this link is helpful.

Thanks!


2016-05-20 11:18 GMT+09:00 Sun Rui <sunrise_...@163.com>:

> 1. I don’t think so
> 2. Arrow is for in-memory columnar execution. While cache is for in-memory
> columnar storage
>
> On May 20, 2016, at 10:16, Todd <bit1...@163.com> wrote:
>
> From the official site http://arrow.apache.org/, Apache Arrow is used for
> Columnar In-Memory storage. I have two quick questions:
> 1. Does spark support Apache Arrow?
> 2. When dataframe is cached in memory, the data are saved in columnar
> in-memory style. What is the relationship between this feature and Apache
> Arrow,that is,
> when the data is in Apache Arrow format，does spark still need the effort
> to cache the dataframe in columnar in-memory?
>
>
>

Re: Does spark support Apache Arrow

[Sun Rui]

1. I don’t think so
2. Arrow is for in-memory columnar execution. While cache is for in-memory 
columnar storage
> On May 20, 2016, at 10:16, Todd <bit1...@163.com> wrote:
> 
> From the official site http://arrow.apache.org/, Apache Arrow is used for 
> Columnar In-Memory storage. I have two quick questions:
> 1. Does spark support Apache Arrow?
> 2. When dataframe is cached in memory, the data are saved in columnar 
> in-memory style. What is the relationship between this feature and Apache 
> Arrow,that is,
> when the data is in Apache Arrow format，does spark still need the effort to 
> cache the dataframe in columnar in-memory?

Does spark support Apache Arrow

[Todd]

From the official site http://arrow.apache.org/, Apache Arrow is used for 
Columnar In-Memory storage. I have two quick questions:
1. Does spark support Apache Arrow?
2. When dataframe is cached in memory, the data are saved in columnar in-memory 
style. What is the relationship between this feature and Apache Arrow,that is,
when the data is in Apache Arrow format，does spark still need the effort to 
cache the dataframe in columnar in-memory?