Here's the image that popped into my mind when I heard about this project,
at best it's a motivating example, at worst it's a distraction:
1. Spark reads parquet from wherever into an arrow structure in shared
memory.
2. Spark executor calls into the Python half of pyspark with a handle to
this me
The JVM may be able to do popcount optimization but it's categorically bad
at other vectorization instructions.
On Wed, Feb 24, 2016 at 18:30 Taro L. Saito wrote:
> Thanks for letting me know.
>
> If we need to embed C++ binaries (.so files) inside java,
> snappy-java's approach https://github.co
Arrow doesn't seem to be ready for use yet. I think it's an aspirational
project. I'd watch for announcements soon but I wouldn't try to
incorporate today.
On Fri, Feb 26, 2016 at 2:10 PM Slava B wrote:
> Agree, also looking for such tutorial
>
> On Fri, Feb 26, 2016 at 11:05 AM, Vishnu Viswan
In the abstract (since I haven't written any code), let me see if I can
make an argument for considering "nullable int" and "int" to both be
worthwhile "primitive" types, as opposed to "Nullable" being a
constructed type over the primitive type "int", in the C++ arena.
Let's assume Arrow's use cas
I meant "We probably don't want std::vector>"
On Fri, Feb 26, 2016 at 10:50 PM Leif Walsh wrote:
> In the abstract (since I haven't written any code), let me see if I can
> make an argument for considering "nullable int" and "int" to both
Seems to me IPC/LPC/RPC focuses on the wrong distinction. I think the right
one is between async message-passing (over a socket), where the receiver
decides when to handle the message, and synchronous/direct memory
manipulation (shared mmap, rdma), where the "client" manipulates the
"server's" (rat
I agree, as a C++ library it is acceptable to let these exceptions bubble
up to the client, and for the C bindings, all exceptions should be caught
and translated to appropriate error codes. Most other languages that
interface with it will probably use the C wrapper and will gain visibility
into t
I am also interested in this.
On Tue, Jun 7, 2016 at 17:37 Holden Karau wrote:
> Hi Everyone,
>
> I'm looking to help get started with Arrow & Spark and to that end I'd like
> to start with getting the Java implementation closer to the spec / C
> implementation. I'm wondering what places people k
+1 this sounds pretty sane
On Fri, Dec 30, 2016 at 06:02 Uwe L. Korn wrote:
> I just had a look over the Apache Calcite approach and I like it very
> much. Both, from a technical and the structural (i.e. keeping the
> website in the main repo). This will enable us to have the format spec
> on Git
I also support the idea of creating an "apache commons modern c++" style
library, maybe tailored toward the needs of columnar data processing
tools. I think APR is the wrong project but I think that *style* of
project is the right direction to aim.
I agree this adds test and release process compl
aused by patch in $COMMON
> > * Arrow proposes patch to $COMMON
> > * ...
> >
> > This is the worst case scenario, of course, but I actually think it is
> > good because it would indicate that the unit testing in $COMMON needs
> > to be improved. Unit testing in
I think Wes' idea that major versions indicate stability of the spec and
minor versions indicate stability of each implementation's API makes sense.
With that in mind, maybe before 1.0 of the spec we should just establish,
within each of the reference language implementations, a mechanism for
speci
Hi all,
I’ve been doing some work lately with Spark’s ML interfaces, which include
sparse and dense Vector and Matrix types, backed on the Scala side by
Breeze. Using these interfaces, you can construct DataFrames whose column
types are vectors and matrices, and though the API isn’t terribly rich,
Matrix are
> not "first class" types in Spark SQL. Spark ML implements them as UDT
> (user-defined types) so it's not clear how to make Spark/Arrow converter
> work with them.
>
> I wonder if Bryan and Holden have some more thoughts on that?
>
> Li
>
> On M
class" types in Spark SQL. Spark ML implements them as UDT
> > (user-defined types) so it's not clear how to make Spark/Arrow converter
> > work with them.
> >
> > I wonder if Bryan and Holden have some more thoughts on that?
> >
> > Li
> >
> >
t
> > of schema metadata or a required part of the schema itself?
> >
> > I feel having it be required might be too restrictive for interop with
> > other systems.
> >
> > On Mon, Apr 9, 2018 at 9:13 PM, Leif Walsh wrote:
> >
> >> My gut feeling is t
[
https://issues.apache.org/jira/browse/ARROW-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558447#comment-15558447
]
Leif Walsh commented on ARROW-189:
--
Do you also want to remove the ability to use the
[
https://issues.apache.org/jira/browse/ARROW-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558446#comment-15558446
]
Leif Walsh commented on ARROW-189:
--
I can take this, I have it working, just nee
[
https://issues.apache.org/jira/browse/ARROW-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558671#comment-15558671
]
Leif Walsh commented on ARROW-189:
--
What is "the right thing"? How can I
[
https://issues.apache.org/jira/browse/ARROW-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558688#comment-15558688
]
Leif Walsh commented on ARROW-189:
--
Right, so I'd delete that if I didn'
[
https://issues.apache.org/jira/browse/ARROW-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558695#comment-15558695
]
Leif Walsh commented on ARROW-189:
--
Oh, I see, you still want downstream packagers t
[
https://issues.apache.org/jira/browse/ARROW-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558850#comment-15558850
]
Leif Walsh commented on ARROW-112:
--
I've done most of this I think, did you als
[
https://issues.apache.org/jira/browse/ARROW-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15558882#comment-15558882
]
Leif Walsh commented on ARROW-112:
--
Ok, cool. Glad I put them in separate commits
[
https://issues.apache.org/jira/browse/ARROW-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563959#comment-15563959
]
Leif Walsh commented on ARROW-112:
--
https://github.com/apache/arrow/pull/168
[
https://issues.apache.org/jira/browse/ARROW-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563958#comment-15563958
]
Leif Walsh commented on ARROW-189:
--
https://github.com/apache/arrow/pull/167
> C
[
https://issues.apache.org/jira/browse/ARROW-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567513#comment-15567513
]
Leif Walsh commented on ARROW-317:
--
How does this relate to ARROW-33?
> [C++] Im
[
https://issues.apache.org/jira/browse/ARROW-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15757648#comment-15757648
]
Leif Walsh commented on ARROW-379:
--
Did anyone add a conda-forge package
[
https://issues.apache.org/jira/browse/ARROW-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758348#comment-15758348
]
Leif Walsh commented on ARROW-379:
--
Perfect, thanks.
> Python: Use setupto
[
https://issues.apache.org/jira/browse/ARROW-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15758361#comment-15758361
]
Leif Walsh commented on ARROW-379:
--
Hmm, maybe not perfect. With the pyarrow-feeds
Leif Walsh created ARROW-805:
Summary: listing empty HDFS directory returns an error instead of
returning empty list
Key: ARROW-805
URL: https://issues.apache.org/jira/browse/ARROW-805
Project: Apache
Leif Walsh created ARROW-2403:
-
Summary: [C++] arrow::CpuInfo::model_name_ destructed twice on exit
Key: ARROW-2403
URL: https://issues.apache.org/jira/browse/ARROW-2403
Project: Apache Arrow
31 matches
Mail list logo
