[jira] [Updated] (ARROW-9513) [Java] Improve documentation in regards to basic-usage / memory-management
[
https://issues.apache.org/jira/browse/ARROW-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sascha schnug updated ARROW-9513:
-
Description:
I'm experimenting with Arrow using Java, C+ +and Python+ IPC format
(Bytestream, File) and Parquet: I am struggling alot on the Java-side, even
after looking for external resources and some code-reading within the
dev-repository .
Observing the state of the documentation, there seems to be a strong favour in
regards to C++ and Python, which is not surprising. The Java part however, is
hard to work with (at least for me; it might be possible that i'm the problem
though). Sadly the Java interface is also the one, which is the most diverging
from what people would usually do in Java.
Acknowledging the user-guide like documentation from
[repo/java|https://github.com/apache/arrow/tree/master/java#getting-started]
(-i don't think this is referenced in the docs and it might only be referenced
by the java-part of the repository -> looks "hidden" as the Java-link in the
docs points to Javadoc-based content- -> known issue: ARROW-9364 ) and it's
warnings about VectorSchemaRoot being special and temporary and also reading
[this external article|https://www.infoq.com/articles/apache-arrow-java]
which also talks about manual memory-management i'm still struggling with a
very simple use-case:
- create and fill VectorSchemaRoot
- write VectorSchemaRoot in IPC format to disk
- read VectorSchemaRoot from IPC format from disk
- INTO some out of scope object not owned by the reader!
I won't put example code here, but refer to my StackOverflow question showing
the problem of mine:
[StackOverflow|https://stackoverflow.com/q/62938237/2320035]
Something about memory-ownership is not working as expected for me.
No matter what tests (dev-repo) or article (e.g. the second link above) i
read, their examples did not help me here as those all are *processing* the
data read in *within the reader-scope* (mostly simple elementwise check), while
i want to read into some *global* object which outlives the reader-object (see
my code on SO or the second link: printing out read data works as long as the
reader is open).
The article above also says:
{code:java}
A vector is managed by one allocator. We say that the allocator owns the
buffer backing the vector. Vector ownership can be transferred from one
allocator to another.
{code}
But how exactly would i populate an empty VectorSchemaRoot (of my class) with
whatever i read in, surviving closing the reader? I experiment with VectorLoad
and VectorUnload, including usage of the only call i found which has
"ownership" in his docstring (batch.cloneWithTransfer), but no success. And
even if working, the Java-based RecordBatch
[link|https://arrow.apache.org/docs/java/] which would be the one using for
this looks completely different then what Pythons does look like
[link|https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.htm]).
Should i be able to see my problem given the documentation? Is there anything
else to read? (I know that there must be in this regards within some Flight /
Gandiva project-code, but i did not find it yet).
Or would it be completely wrong to keep VectorSchemaRoot as core-object to
handle all my data?
Feel free to close this issue if you think, that documentation is *not*
incomplete.
Thanks,
Sascha
was:
I'm experimenting with Arrow using Java, C+ +and Python+ IPC format
(Bytestream, File) and Parquet: I am struggling alot on the Java-side, even
after looking for external resources and some code-reading within the
dev-repository .
Observing the state of the documentation, there seems to be a strong favour in
regards to C++ and Python, which is not surprising. The Java part however, is
hard to work with (at least for me; it might be possible that i'm the problem
though). Sadly the Java interface is also the one, which is the most diverging
from what people would usually do in Java.
Acknowledging the user-guide like documentation from
[repo/java|https://github.com/apache/arrow/tree/master/java#getting-started] (i
don't think this is referenced in the docs and it might only be referenced by
the java-part of the repository -> looks "hidden" as the Java-link in the docs
points to Javadoc-based content) and it's warnings about VectorSchemaRoot being
special and temporary and also reading [this external
article|https://www.infoq.com/articles/apache-arrow-java]
which also talks about manual memory-management i'm still struggling with a
very simple use-case:
- create and fill VectorSchemaRoot
- write VectorSchemaRoot in IPC format to disk
- read VectorSchemaRoot from IPC format from disk
- INTO some out of scope object not owned by the reader!
I won't put example code here, but refer to my StackOverflow question showing
the
[jira] [Updated] (ARROW-9513) [Java] Improve documentation in regards to basic-usage / memory-management
[
https://issues.apache.org/jira/browse/ARROW-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sascha schnug updated ARROW-9513:
-
Description:
I'm experimenting with Arrow using Java, C+ +and Python+ IPC format
(Bytestream, File) and Parquet: I am struggling alot on the Java-side, even
after looking for external resources and some code-reading within the
dev-repository .
Observing the state of the documentation, there seems to be a strong favour in
regards to C++ and Python, which is not surprising. The Java part however, is
hard to work with (at least for me; it might be possible that i'm the problem
though). Sadly the Java interface is also the one, which is the most diverging
from what people would usually do in Java.
Acknowledging the user-guide like documentation from
[repo/java|https://github.com/apache/arrow/tree/master/java#getting-started]
(-i don't think this is referenced in the docs and it might only be referenced
by the java-part of the repository > looks "hidden" as the Java-link in the
docs points to Javadoc-based content- -> known issue: ARROW-9364 ) and it's
warnings about VectorSchemaRoot being special and temporary and also reading
[this external article|https://www.infoq.com/articles/apache-arrow-java]
which also talks about manual memory-management i'm still struggling with a
very simple use-case:
- create and fill VectorSchemaRoot
- write VectorSchemaRoot in IPC format to disk
- read VectorSchemaRoot from IPC format from disk
- INTO some out of scope object not owned by the reader!
I won't put example code here, but refer to my StackOverflow question showing
the problem of mine:
[StackOverflow|https://stackoverflow.com/q/62938237/2320035]
Something about memory-ownership is not working as expected for me.
No matter what tests (dev-repo) or article (e.g. the second link above) i
read, their examples did not help me here as those all are *processing* the
data read in *within the reader-scope* (mostly simple elementwise check), while
i want to read into some *global* object which outlives the reader-object (see
my code on SO or the second link: printing out read data works as long as the
reader is open).
The article above also says:
{code:java}
A vector is managed by one allocator. We say that the allocator owns the
buffer backing the vector. Vector ownership can be transferred from one
allocator to another.
{code}
But how exactly would i populate an empty VectorSchemaRoot (of my class) with
whatever i read in, surviving closing the reader? I experiment with VectorLoad
and VectorUnload, including usage of the only call i found which has
"ownership" in his docstring (batch.cloneWithTransfer), but no success. And
even if working, the Java-based RecordBatch
[link|https://arrow.apache.org/docs/java/] which would be the one using for
this looks completely different then what Pythons does look like
[link|https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.htm]).
Should i be able to see my problem given the documentation? Is there anything
else to read? (I know that there must be in this regards within some Flight /
Gandiva project-code, but i did not find it yet).
Or would it be completely wrong to keep VectorSchemaRoot as core-object to
handle all my data?
Feel free to close this issue if you think, that documentation is *not*
incomplete.
Thanks,
Sascha
was:
I'm experimenting with Arrow using Java, C+ +and Python+ IPC format
(Bytestream, File) and Parquet: I am struggling alot on the Java-side, even
after looking for external resources and some code-reading within the
dev-repository .
Observing the state of the documentation, there seems to be a strong favour in
regards to C++ and Python, which is not surprising. The Java part however, is
hard to work with (at least for me; it might be possible that i'm the problem
though). Sadly the Java interface is also the one, which is the most diverging
from what people would usually do in Java.
Acknowledging the user-guide like documentation from
[repo/java|https://github.com/apache/arrow/tree/master/java#getting-started]
(-i don't think this is referenced in the docs and it might only be referenced
by the java-part of the repository -> looks "hidden" as the Java-link in the
docs points to Javadoc-based content- -> known issue: ARROW-9364 ) and it's
warnings about VectorSchemaRoot being special and temporary and also reading
[this external article|https://www.infoq.com/articles/apache-arrow-java]
which also talks about manual memory-management i'm still struggling with a
very simple use-case:
- create and fill VectorSchemaRoot
- write VectorSchemaRoot in IPC format to disk
- read VectorSchemaRoot from IPC format from disk
- INTO some out of scope object not owned by the reader!
I won't put example code here, but refer to my
[jira] [Updated] (ARROW-9513) [Java] Improve documentation in regards to basic-usage / memory-management
[
https://issues.apache.org/jira/browse/ARROW-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sascha schnug updated ARROW-9513:
-
Description:
I'm experimenting with Arrow using Java, C+ +and Python+ IPC format
(Bytestream, File) and Parquet: I am struggling alot on the Java-side, even
after looking for external resources and some code-reading within the
dev-repository .
Observing the state of the documentation, there seems to be a strong favour in
regards to C++ and Python, which is not surprising. The Java part however, is
hard to work with (at least for me; it might be possible that i'm the problem
though). Sadly the Java interface is also the one, which is the most diverging
from what people would usually do in Java.
Acknowledging the user-guide like documentation from
[repo/java|https://github.com/apache/arrow/tree/master/java#getting-started] (i
don't think this is referenced in the docs and it might only be referenced by
the java-part of the repository -> looks "hidden" as the Java-link in the docs
points to Javadoc-based content) and it's warnings about VectorSchemaRoot being
special and temporary and also reading [this external
article|https://www.infoq.com/articles/apache-arrow-java]
which also talks about manual memory-management i'm still struggling with a
very simple use-case:
- create and fill VectorSchemaRoot
- write VectorSchemaRoot in IPC format to disk
- read VectorSchemaRoot from IPC format from disk
- INTO some out of scope object not owned by the reader!
I won't put example code here, but refer to my StackOverflow question showing
the problem of mine:
[StackOverflow|https://stackoverflow.com/q/62938237/2320035]
Something about memory-ownership is not working as expected for me.
No matter what tests (dev-repo) or article (e.g. the second link above) i
read, their examples did not help me here as those all are *processing* the
data read in *within the reader-scope* (mostly simple elementwise check), while
i want to read into some *global* object which outlives the reader-object (see
my code on SO or the second link: printing out read data works as long as the
reader is open).
The article above also says:
{code:java}
A vector is managed by one allocator. We say that the allocator owns the
buffer backing the vector. Vector ownership can be transferred from one
allocator to another.
{code}
But how exactly would i populate an empty VectorSchemaRoot (of my class) with
whatever i read in, surviving closing the reader? I experiment with VectorLoad
and VectorUnload, including usage of the only call i found which has
"ownership" in his docstring (batch.cloneWithTransfer), but no success. And
even if working, the Java-based RecordBatch
[link|https://arrow.apache.org/docs/java/] which would be the one using for
this looks completely different then what Pythons does look like
[link|https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.htm]).
Should i be able to see my problem given the documentation? Is there anything
else to read? (I know that there must be in this regards within some Flight /
Gandiva project-code, but i did not find it yet).
Or would it be completely wrong to keep VectorSchemaRoot as core-object to
handle all my data?
Feel free to close this issue if you think, that documentation is *not*
incomplete.
Thanks,
Sascha
was:
I'm experimenting with Arrow using Java, C++ and Python + IPC format
(Bytestream, File) and Parquet: I am struggling alot on the Java-side, even
after looking for external resources and some code-reading within the
dev-repository .
Observing the state of the documentation, there seems to be a strong favour in
regards to C++ and Python, which is not surprising. The Java part however, is
hard to work with (at least for me; it might be possible that i'm the problem
though). Sadly the Java interface is also the one, which is the most diverging
from what people would usually do in Java.
Acknowledging the user-guide like documentation from
[repo/java|https://github.com/apache/arrow/tree/master/java#getting-started] (i
don't think this is referenced in the docs and it might only be referenced by
the java-part of the repository -> looks "hidden" as the Java-link in the docs
points to Javadoc-based content) and it's warnings about VectorSchemaRoot being
special and temporary and also reading [this external
article|https://www.infoq.com/articles/apache-arrow-java]
which also talks about manual memory-management i'm still struggling with a
very simple use-case:
- create and fill VectorSchemaRoot
- write VectorSchemaRoot in IPC format to disk
- read VectorSchemaRoot from IPC format from disk
- INTO some out of scope object not owned by the reader!
I won't put example code here, but refer to my StackOverflow question showing
the problem of mine:
