bump. I have same question at Petr. SPARK-13534 seem to only solve de(serialization) issue involved between rdd and python objects. However, isn't Arrow can be standard for in-memory columnar representation? may be alternative to spark current in-memory store (k-v blocks or tungsten)
Thanks Nir On Wed, Feb 24, 2016 at 3:56 AM, Petr Novak <oss.mli...@gmail.com> wrote: > > How Arrows collide with Tungsten and its binary in-memory format. It will > still has to convert between them. I assume they use similar > concepts/layout hence it is likely the conversion can be quite efficient. > Or is there a change that the current Tungsten in memory format would be > replaced by Arrows in the future. The same applies for Impala, Drill and > all others. Is the goal to unify internal in-memory representation for all > of them or the benefit is going to be in conversions faster by e.g. order > of magnitude? > > Many thanks for any explanation, > Petr > -- [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] <https://twitter.com/Xactly> [image: Facebook] <https://www.facebook.com/XactlyCorp> [image: YouTube] <http://www.youtube.com/xactlycorporation>