[jira] [Created] (ARROW-1469) Segfault when serialize Pandas series with mixed object type

2017-09-06 Thread GUAN Hao (JIRA)
GUAN Hao created ARROW-1469: --- Summary: Segfault when serialize Pandas series with mixed object type Key: ARROW-1469 URL: https://issues.apache.org/jira/browse/ARROW-1469 Project: Apache Arrow Issu

[jira] [Created] (ARROW-1470) [C++] Add BufferAllocator abstract interface

2017-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-1470: --- Summary: [C++] Add BufferAllocator abstract interface Key: ARROW-1470 URL: https://issues.apache.org/jira/browse/ARROW-1470 Project: Apache Arrow Issue Type: N

[jira] [Created] (ARROW-1471) [JAVA] Document requirements and non/requirements for ValueVector updates

2017-09-06 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created ARROW-1471: - Summary: [JAVA] Document requirements and non/requirements for ValueVector updates Key: ARROW-1471 URL: https://issues.apache.org/jira/browse/ARROW-1471 Project: Ap

[jira] [Created] (ARROW-1473) [JAVA] Create Prototype Code Hierarchy (alt A)

2017-09-06 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created ARROW-1473: - Summary: [JAVA] Create Prototype Code Hierarchy (alt A) Key: ARROW-1473 URL: https://issues.apache.org/jira/browse/ARROW-1473 Project: Apache Arrow Issue T

[jira] [Created] (ARROW-1472) [JAVA] Design updated ValueVector Object Hierarchy

2017-09-06 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created ARROW-1472: - Summary: [JAVA] Design updated ValueVector Object Hierarchy Key: ARROW-1472 URL: https://issues.apache.org/jira/browse/ARROW-1472 Project: Apache Arrow Iss

[jira] [Created] (ARROW-1474) [JAVA] Create Prototype Code Hierarchy (alt B)

2017-09-06 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created ARROW-1474: - Summary: [JAVA] Create Prototype Code Hierarchy (alt B) Key: ARROW-1474 URL: https://issues.apache.org/jira/browse/ARROW-1474 Project: Apache Arrow Issue T

[jira] [Created] (ARROW-1475) [JAVA] Create Benchmarking Suite for prototypes

2017-09-06 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created ARROW-1475: - Summary: [JAVA] Create Benchmarking Suite for prototypes Key: ARROW-1475 URL: https://issues.apache.org/jira/browse/ARROW-1475 Project: Apache Arrow Issue

[jira] [Created] (ARROW-1477) Create Benchmarking Suite for final ValueVector updates

2017-09-06 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created ARROW-1477: - Summary: Create Benchmarking Suite for final ValueVector updates Key: ARROW-1477 URL: https://issues.apache.org/jira/browse/ARROW-1477 Project: Apache Arrow

[jira] [Created] (ARROW-1476) [JAVA] Implement final ValueVector updates

2017-09-06 Thread Jacques Nadeau (JIRA)
Jacques Nadeau created ARROW-1476: - Summary: [JAVA] Implement final ValueVector updates Key: ARROW-1476 URL: https://issues.apache.org/jira/browse/ARROW-1476 Project: Apache Arrow Issue Type:

Re: Apache Arrow at JupyterCon

2017-09-06 Thread Wes McKinney
It should be possible to have an ArrowBuf backed by a MappedByteBuffer. Anyone reading is welcome to dig in and write a patch for this. Semantically this is what we have done in C++ -- a memory map inherits from arrow::Buffer, so we can slice and dice a memory map as we would any other Buffer obje

[jira] [Created] (ARROW-1478) clear should release the buffer only if the buffer is not NULL

2017-09-06 Thread SIDDHARTH TEOTIA (JIRA)
SIDDHARTH TEOTIA created ARROW-1478: --- Summary: clear should release the buffer only if the buffer is not NULL Key: ARROW-1478 URL: https://issues.apache.org/jira/browse/ARROW-1478 Project: Apache Ar

Re: Apache Arrow at JupyterCon

2017-09-06 Thread Gary Wong
The ArrowBuf is inherited from AbstractByteBuf, the AbstractByteBuf is defined in the Netty library, it does more like a memory pool not a pure buffer so that's why ArrowBuf is not backed by ByteBuffer as now. I have ever tried to make ArrowBuf build on top of DurableBuffer of Mnemonic, but looks

[jira] [Created] (ARROW-1479) [JS] Expand JavaScript implementation

2017-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-1479: --- Summary: [JS] Expand JavaScript implementation Key: ARROW-1479 URL: https://issues.apache.org/jira/browse/ARROW-1479 Project: Apache Arrow Issue Type: New Feat

[jira] [Created] (ARROW-1480) [Python] Improve performance of serializing sets

2017-09-06 Thread Philipp Moritz (JIRA)
Philipp Moritz created ARROW-1480: - Summary: [Python] Improve performance of serializing sets Key: ARROW-1480 URL: https://issues.apache.org/jira/browse/ARROW-1480 Project: Apache Arrow Issue

Re: Apache Arrow at JupyterCon

2017-09-06 Thread Wes McKinney
Thanks Gary, that is helpful context. In light if this, it might be worth writing some kind of a proposal for how to enable the Java vector classes to be backed by some other kind of byte buffers. It might be that an alternative version of portions of the Arrow Java library (i.e. decoupled from Net

Re: Apache Arrow at JupyterCon

2017-09-06 Thread Jacques Nadeau
This is a interesting problem but also pretty complex. Arrow's Java memory management model is complex on purpose (see https://github.com/apache/arrow/blob/master/java/memory/src/main/java/org/apache/arrow/memory/README.md for more info). It is designed to reserve and share memory in multiple hiera