[jira] [Created] (ARROW-3195) [C++] NumPy initialization error check is missing in test

Kouhei Sutou created ARROW-3195: --- Summary: [C++] NumPy initialization error check is missing in test Key: ARROW-3195 URL: https://issues.apache.org/jira/browse/ARROW-3195 Project: Apache Arrow

[jira] [Created] (ARROW-3194) Fix setValueCount in spitAndTransfer for variable width vectors

Siddharth Teotia created ARROW-3194: --- Summary: Fix setValueCount in spitAndTransfer for variable width vectors Key: ARROW-3194 URL: https://issues.apache.org/jira/browse/ARROW-3194 Project: Apache

Re: [JAVA] Supporting zero copy arrow-vector

Thanks. That's crystal clear for me now. On Fri, Sep 7, 2018 at 1:16 PM Jacques Nadeau wrote: > I opened a jira to describe what I think needs to be done here. Check it > out: > > https://issues.apache.org/jira/browse/ARROW-3191 > > > On Fri, Sep 7, 2018 at 10:47 AM Wes McKinney wrote: > > >

[jira] [Created] (ARROW-3192) [Java] Implement "ArrowBufReadChannel" abstraction and alternate MessageSerializer that uses this

Wes McKinney created ARROW-3192: --- Summary: [Java] Implement "ArrowBufReadChannel" abstraction and alternate MessageSerializer that uses this Key: ARROW-3192 URL: https://issues.apache.org/jira/browse/ARROW-3192

Re: [JAVA] Supporting zero copy arrow-vector

I opened a jira to describe what I think needs to be done here. Check it out: https://issues.apache.org/jira/browse/ARROW-3191 On Fri, Sep 7, 2018 at 10:47 AM Wes McKinney wrote: > Seems like you should be able to construct an UnsafeDirectByteBuf from > a MappedByteBuffer, and then wrap that

[jira] [Created] (ARROW-3191) [Java] Add support for ArrowBuf to point to arbitrary memory.

Jacques Nadeau created ARROW-3191: - Summary: [Java] Add support for ArrowBuf to point to arbitrary memory. Key: ARROW-3191 URL: https://issues.apache.org/jira/browse/ARROW-3191 Project: Apache Arrow

Re: Buffer writers and seek method, NativeFile.is_seekable proposal

hi Paul, We aren't talking about columnar data structures, but file interfaces, i.e. the C++ classes in https://github.com/apache/arrow/tree/master/cpp/src/arrow/io - Wes On Fri, Sep 7, 2018 at 2:56 PM Paul Rogers wrote: > > Hi Wes, > > Intersting. Random-access writes is easy for fixed-width

Re: Buffer writers and seek method, NativeFile.is_seekable proposal

Hi Wes, Intersting. Random-access writes is easy for fixed-width vectors. I'm curious how it might be done for variable-width vectors (VARCHAR, or arrays) given the structure of the offset vectors? Is the structure of the offset vector changing (to include, say, the start and length of each

Re: Buffer writers and seek method, NativeFile.is_seekable proposal

I just created https://issues.apache.org/jira/browse/ARROW-3189 On Fri, Sep 7, 2018 at 2:39 PM Wes McKinney wrote: > > hi Pearu, > > Sounds good to me. I'd always intended to add support for random > access writes but have not done it yet. > > Thanks, > Wes > On Fri, Sep 7, 2018 at 3:51 AM Pearu

[jira] [Created] (ARROW-3190) [C++] "WriteableFile" is misspelled, should be renamed "WritableFile" with deprecation for old name

Wes McKinney created ARROW-3190: --- Summary: [C++] "WriteableFile" is misspelled, should be renamed "WritableFile" with deprecation for old name Key: ARROW-3190 URL: https://issues.apache.org/jira/browse/ARROW-3190

[jira] [Created] (ARROW-3189) [Python] Support seek(...) on writable files that support it

Wes McKinney created ARROW-3189: --- Summary: [Python] Support seek(...) on writable files that support it Key: ARROW-3189 URL: https://issues.apache.org/jira/browse/ARROW-3189 Project: Apache Arrow

Re: Buffer writers and seek method, NativeFile.is_seekable proposal

hi Pearu, Sounds good to me. I'd always intended to add support for random access writes but have not done it yet. Thanks, Wes On Fri, Sep 7, 2018 at 3:51 AM Pearu Peterson wrote: > > Hi, > > In Arrow C++, various buffer writers define Seek method while in > pyarrow the seek is defined only for

Re: spectrum.chat as community channel

Thanks Dimitri -- let's keep an eye on this. I'm quite concerned about anything that seems like "chat". The risk is that we end up with lower-quality discourse and that actionable development items do not end up in JIRA -- that's what was happening on Slack. If everyone were super proactive about

Re: [JAVA] Supporting zero copy arrow-vector

Seems like you should be able to construct an UnsafeDirectByteBuf from a MappedByteBuffer, and then wrap that with UnsafeDirectLittleEndian to get zero-copy access to a memory map. Does that sound right?

Re: [JAVA] Supporting zero copy arrow-vector

Interesting, so basically I can still use the public constructor public ArrowBuf(AtomicInteger refCnt, BufferLedger ledger, UnsafeDirectLittleEndian byteBuf, BufferManager manager, ArrowByteBufAllocator alloc, int offset, int length, boolean isEmpty) Instead, override

spectrum.chat as community channel

Hi everybody, I wanted to bring spectrum.chat (https://spectrum.chat) to your attention. It is a community communication platform which seeks to combine the advantages of mailing lists (searchable, easily accessible) with the interactivity of chats. A spectrum community contains multiple

Re: Lighter build matrix on a language specific fork.

The rpath issue does not need to be resolved right now, though. I will review the patch again and merge if all looks good On Fri, Sep 7, 2018 at 4:53 AM Kouhei Sutou wrote: > Hi, > > We can get Arrow's library directory by > "pkg-config --variable=libdir arrow". Does this help this case? > >

Re: [JAVA] Supporting zero copy arrow-vector

It is on purpose that the ArrowBuf is final. It is done to ensure a single impl and performance reasons. ArrowBuf is primarily a memory address and a length and wants zero indirection to the reading/writing of that. It does, however, wrap several types of substructures as long as they have that

[jira] [Created] (ARROW-3188) [Python] Table.from_arrays segfaults if lists and schema are passed

Krisztian Szucs created ARROW-3188: -- Summary: [Python] Table.from_arrays segfaults if lists and schema are passed Key: ARROW-3188 URL: https://issues.apache.org/jira/browse/ARROW-3188 Project:

[jira] [Created] (ARROW-3187) [Plasma] Change Logging to glog

Yuhong Guo created ARROW-3187: - Summary: [Plasma] Change Logging to glog Key: ARROW-3187 URL: https://issues.apache.org/jira/browse/ARROW-3187 Project: Apache Arrow Issue Type: New Feature

Re: Lighter build matrix on a language specific fork.

Hi, We can get Arrow's library directory by "pkg-config --variable=libdir arrow". Does this help this case? Thanks, -- kou In <4712da01-7d10-44aa-971b-e55524197...@purrple.cat> "Re: Lighter build matrix on a language specific fork. " on Fri, 7 Sep 2018 10:47:03 +0200, Romain Francois

Re: Lighter build matrix on a language specific fork.

Addressed most of the comments. Anyone knows how to set rpath dynamically in: https://github.com/apache/arrow/pull/2489#discussion_r215875597 Maybe pkg-config can help ? > Le 7 sept. 2018 à 00:40, Wes McKinney a écrit : > >

Buffer writers and seek method, NativeFile.is_seekable proposal

Hi, In Arrow C++, various buffer writers define Seek method while in pyarrow the seek is defined only for buffer readers (for instance, NativeFile.seek references only rd_file). So, pyarrow relates 'seekable' strictly to 'readable' file property while 'seekable' would make sense also when a file