[ https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935497#comment-16935497 ]
runzhiwang edited comment on LIVY-667 at 9/23/19 1:21 AM: ---------------------------------------------------------- [~mgaido] Because of the limited memory and huge size of user data, increasing memory cannot solve this problem. Otherwise, a lot of drivers with large memory will exhaust the cluster memory very soon. was (Author: runzhiwang): [~mgaido] Because of the limited memory and huge size of user data, increasing memory cannot solve this problem. Otherwise, a lot of drivers with large memory will exist the cluster memory very soon. > Support query a lot of data. > ---------------------------- > > Key: LIVY-667 > URL: https://issues.apache.org/jira/browse/LIVY-667 > Project: Livy > Issue Type: Bug > Components: Thriftserver > Affects Versions: 0.6.0 > Reporter: runzhiwang > Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time instead of the whole rdd to avoid > OutOfMemory. However, if the largest partition is too big, the OutOfMemory > still occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)