[
https://issues.apache.org/jira/browse/ARROW-10853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248739#comment-17248739
]
Liya Fan commented on ARROW-10853:
----------------------------------
[~uwe] Thanks a lot for your feedback.
By setting JdbcToArrowConfig#targetBatchSize to NO_LIMIT_BATCH_SIZE (-1), we
get everything in a single batch. So there is no need to iterate over the
dataset. (However, the caller must make sure there is enough memory to avoid
OOM).
For the interface issue, currently the process works in two steps:
1. {{(connection, query) -> result set}}, and
2. {{(result set, config) -> VectorSchemaRoot}}.
The above two steps are separate, primarily because they are independent
processes. In particular, the functionality of step 2 is provided by Arrow,
whereas step 1 does not have much to do with Arrow.
> [Java] Undeprecate sqlToArrow helpers
> -------------------------------------
>
> Key: ARROW-10853
> URL: https://issues.apache.org/jira/browse/ARROW-10853
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java
> Affects Versions: 2.0.0
> Reporter: Uwe Korn
> Assignee: Uwe Korn
> Priority: Major
> Fix For: 3.0.0
>
>
> These helper functions are really useful when called from Python as they deal
> with a lot of "internals" of Java that we don't want to handle from the
> Python side. We rather would keep using these functions.
> Note that some of them are broken due to recent refactoring and only return
> 1024 rows (the default iterator size) without the ability to change that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
