Poll question: why did you choose Arrow?
Personally: I researched Arrow because it's a spec for IPC. (My requirement
was: "wrap computations in a separate process.") I chose Arrow for its
community and ecosystem -- in other words, because my peers chose it.
I happen to use the compute kernel and Parquet capabilities every day; but
they did not sway me at all. I would choose Arrow if it were nothing but
this spec and this community. (I chose HTML, after all.)
I see the *code* as one enormous proof that the *spec* is good, and as a
collection of examples and best practices.
... so a great pitch to me would be: "Apache Arrow is a data format and
toolbox for efficient in-memory processing."
Enjoy life,
Adam
On Tue, May 18, 2021 at 2:38 AM Aldrin <akmon...@ucsc.edu.invalid> wrote:
"Apache Arrow is a data processing library that also provides a uniform,
efficient interface for data systems."
This probably still isn't quite right, I imagine the bit about "for data
systems" needs some addition (maybe "for transport between data systems")?
My primary motivators:
- "A data processing library":
- Arrow provides many language bindings, but ultimately they're all
part of the same "library ecosystem", which I think is fine to
capture in
"library"
- A main goal of arrow is for processing to be fast, whatever that
processing may be
- "uniform, efficient interface for data systems":
- Arrow, provides (or tries to) a cohesive ("uniform") interface for
data processing (although it has several APIs to do this)
- Also, IMO, a motivation for arrow was a format and library to
facilitate processing, but that provided functions and
interfaces to easily
translate into optimized data formats used by disparate data systems
(cassandra, hadoop, etc.).
- Arrow tries to be transparently zero-copy, which is part of the
interface for efficiency
- Arrow certainly has a data format, but that format is the crux of the
interface (IMO). However, it also makes using other formats easy (via
filesystem API and parquet reader/writers, etc.). So, focusing on the
data
format seems unnecessary in such a terse description.
Aldrin Montana
Computer Science PhD Student
UC Santa Cruz
On Mon, May 17, 2021 at 5:07 PM Weston Pace <weston.p...@gmail.com> wrote:
I'd avoid the word "structured" as it is somewhat ill-defined.
On Mon, May 17, 2021 at 12:37 PM Mauricio Vargas
<mauri...@ursacomputing.com> wrote:
more marketed:
How about: "Apache Arrow is a format and language-agnostic library
focused
on efficient sharing and processing of structured data."
On Mon, May 17, 2021 at 6:25 PM Micah Kornfield <emkornfi...@gmail.com
wrote:
How about: "Apache Arrow is a collection of specifications, cross
language
libraries and applications focused on efficient sharing and
processing
of
structured data."
On Mon, May 17, 2021 at 3:06 PM Wes McKinney <wesmck...@gmail.com>
wrote:
On Mon, May 17, 2021 at 4:58 PM Weston Pace <weston.p...@gmail.com
wrote:
“Apache Arrow is a format and compute kernel for in-memory
data”
I like this but no one ever knows what "in-memory" means (or they
just
think 'data is always in memory'). How about...
"Apache Arrow is a format and compute kernel for zero-copy
processing
and sharing of data."
or...
"Apache Arrow is a format and compute kernel for processing and
sharing data without serialization overhead."
A few issues with this:
* Multiple PL aspect unclear (is a single piece of software, or
multiple pieces of software?)
* Development platform aspect unclear
I see that some people don't like the word "platform". Some people
come to this project and want to find an end-to-end application,
rather than a developer toolkit that they can use to build
applications. Perhaps we should be more explicit and use
"computational development toolkit" instead of "platform".
Although marshalling[1] would probably be a more precise word it
is
not as well known.
[1] https://en.wikipedia.org/wiki/Marshalling_(computer_science)
On Mon, May 17, 2021 at 9:36 AM Mauricio Vargas
<mauri...@ursacomputing.com> wrote:
a few ideas
github.com/apache/arrow - Apache Arrow is an efficient library
for
big data
processing and sharing
github.com/apache/arrow - Apache Arrow is a computational tool
for
processing, storing and sharing large datasets
github.com/apache/arrow - Apache Arrow is a fast and simple
library
for
big data analytics
*github.com/apache/arrow <http://github.com/apache/arrow> -
Apache
Arrow is
a powerful workhorse for analytic operations on modern
hardware*
On Mon, May 17, 2021 at 3:13 PM Julian Hyde <
jhyde.apa...@gmail.com>
wrote:
Alright, well, whatever it is, it must fit into one breath.
If
the
high-concept pitch is successful, people will stick around
for
the
full
pitch.
Words such as “platform” and “enable” are noise. You say
“platform”,
they
start to say “what exactly do you mean by platform”, the
elevator
doors
open, and they’re gone.
“Apache Arrow is a format and compute kernel for in-memory
data”
On May 17, 2021, at 12:03 PM, Eduardo Ponce <
edponc...@gmail.com
wrote:
One more suggestion for the bucket:
"Apache Arrow is a computational platform for efficient
in-memory
data
representation and processing."
On Mon, May 17, 2021 at 2:49 PM Wes McKinney <
wesmck...@gmail.com>
wrote:
I think less is better in the description, but
unfortunately the
association of Arrow as being "just a data format" has
been
actively
harmful in some ways to community growth. We have a data
format,
yes,
but we are also creating a computational platform to go
hand-in-hand
with the data format to make it easier to build fast
applications
that
use the data format. So the description needs to capture
both of
these
ideas.
On Mon, May 17, 2021 at 12:15 PM Julian Hyde <
jhyde.apa...@gmail.com>
wrote:
I think that the “cross-language development platform
for”
is
noise.
(I’m sure that JPEG developers think that JPEG is a
“cross-language
development platform” too. But it isn’t. It is an image
format.)
"Apache Arrow is data format for efficient in-memory
processing.”
I’ll note that In marketing speak, we are developing a
high-concept
pitch [1] here. Every company needs a name, a brand, a
high-concept
pitch,
and 3- or 4-sentence description. But every Apache project
needs
these
too.
It’s worth spending the time on the description, also, and
then
use
them in
all the places that we describe Arrow.
Julian
[1]
https://www.growthink.com/content/whats-your-high-concept-pitch
On May 17, 2021, at 7:38 AM, Eduardo Ponce <
edponc...@gmail.com
wrote:
I agree with Nate's and Brian's suggestions, but would
like to
add
that we
can make it a one-liner for more conciseness and
consistency
with
other
Apache projects.
Apologies if it seems I am going around the suggestions
loop
again.
"Apache Arrow is a cross-language development platform
enabling
efficient
in-memory data processing and transport."
On Mon, May 17, 2021 at 10:11 AM Brian Hulette <
bhule...@apache.org>
wrote:
Thank you for bringing this up Dominik. I sampled some
of the
descriptions
for other Apache projects I frequent, the ones with a
meaningful
description have a single sentence:
github.com/apache/spark - Apache Spark - A unified
analytics
engine
for
large-scale data processing
github.com/apache/beam - Apache Beam is a unified
programming
model
for
Batch and Streaming
github.com/apache/avro - Apache Avro is a data
serialization
system
Several others (Flink, Hadoop, ...) just have "[Mirror
of]
Apache
<name>"
as the description.
+1 for Nate's suggestion "Apache Arrow is a
cross-language
development
platform for in-memory data. It enables systems to
process
and
transport
data more efficiently."
On Mon, May 17, 2021 at 5:23 AM Wes McKinney <
wesmck...@gmail.com>
wrote:
It's probably best for description to limit mentions
of
specific
features. There are some high level features mentioned
in
the
description now ("computational libraries and
zero-copy
streaming
messaging and interprocess communication"), but now in
2021
since
the
project has grown so much, it could leave people with
a
limited view
of what they might find here.
On Mon, May 17, 2021 at 12:14 AM Mauricio Vargas
<mauri...@ursacomputing.com> wrote:
How about
'Apache Arrow is a cross-language development
platform
for
in-memory
data.
It enables systems to process and transport data
efficiently,
providing a
simple and fast library for partitioning of large
tables'?
Sorry the delay, long election day
On Sun, May 16, 2021, 2:27 PM Nate Bauernfeind <
natebauernfe...@deephaven.io>
wrote:
Suggestion: faster -> more efficiently
"Apache Arrow is a cross-language development
platform for
in-memory
data. It enables systems to process and transport
data
more
efficiently."
On Sun, May 16, 2021 at 11:35 AM Wes McKinney <
wesmck...@gmail.com
wrote:
Here's what there now:
"Apache Arrow is a cross-language development
platform
for
in-memory
data. It specifies a standardized
language-independent
columnar
memory
format for flat and hierarchical data, organized
for
efficient
analytic operations on modern hardware. It also
provides
computational
libraries and zero-copy streaming messaging and
interprocess
communication…"
How about something shorter like
"Apache Arrow is a cross-language development
platform
for
in-memory
data. It enables systems to process and transport
data
faster."
Suggestions / refinements from others welcome
On Sat, May 15, 2021 at 9:12 PM Dominik Moritz <
domor...@cmu.edu
wrote:
Super minor issue but could someone make the
description
on
GitHub
shorter?
GitHub puts the description into the title of the
page
and makes
it
hard
to find it in URL autocomplete.
--
--
Adam Hooper
+1-514-882-9694
http://adamhooper.com
