[
https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935616#comment-16935616
]
Marco Gaido commented on LIVY-667:
----------------------------------
[~runzhiwang] let me cite you in the JIRA body:
> When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator
> to load one partition at each time
this means that if the driver is as big as the executors, there should not be
OOM, since the partition was already held in memory on the executors. And in
general having a driver large as the executors should not be a big deal in
terms on memory occupation on the cluster in percentage, as I expect to have
much more executors than drivers..
> Support query a lot of data.
> ----------------------------
>
> Key: LIVY-667
> URL: https://issues.apache.org/jira/browse/LIVY-667
> Project: Livy
> Issue Type: Bug
> Components: Thriftserver
> Affects Versions: 0.6.0
> Reporter: runzhiwang
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator
> to load one partition at each time instead of the whole rdd to avoid
> OutOfMemory. However, if the largest partition is too big, the OutOfMemory
> still occurs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
