hi Kai, This sounds like it might merit a separate thread to discuss the growth of Arrow as a modular ecosystem of libraries in different programming languages and related tools (we've discussed shared memory data access and metadata representation, but not questions around ownership and management of shared memory resources, which inevitably will come up). Composability around the shared memory layout is an extremely powerful concept, just as zero-copy compatibility with BLAS / LAPACK / Intel MKL is very important for high performance linear algebra in scientific computing use cases.
(Note that I recently have been working to revive Parquet C++ as a standalone library and have become a committer on that project, but my first commit was only in January. More specific comments there might be valuable on the Parquet mailing list.) - Wes On Thu, Mar 17, 2016 at 3:44 PM, Zheng, Kai <kai.zh...@intel.com> wrote: > Sounds good to have all these compatible, modular goals and changes, for > Apache Arrow, in the early stage. > > On the other hand, not being afraid of changes keep evolving towards the core > goals and the well-defined initiatives, which is also important. Parquet is > relevant and also a good example. IMO, it's simply poorly organized. Yes it's > all about history, but as a still young project, it looks like to me it just > stops evolving. Will Arrow be another Parquet? > > Regards, > Kai > > -----Original Message----- > From: Wes McKinney [mailto:w...@cloudera.com] > Sent: Thursday, March 17, 2016 6:03 AM > To: dev@arrow.apache.org > Subject: Re: Understanding "shared" memory implications > > On Wed, Mar 16, 2016 at 2:33 PM, Jacques Nadeau <jacq...@apache.org> wrote: >> >> For Arrow, let's make sure that we do our best to accomplish both (1) >> and (2). They seem like entirely compatible goals. >> >> > > For my part on the C++ side, I plan to proceed with a hub-and-spoke model. A > minimal small core library with "leaf" shared libraries (for > example: Parquet read/write adapter) that you can opt-in to building. > This will add some extra linker configuration complexity for downstream users > but to the benefit of a less monolithic library stack (which I don't think > anyone wants).
