[ https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935829#comment-16935829 ]
Yiheng Wang commented on LIVY-667: ---------------------------------- Hi Marco. I think Spark compute on the partition data through an iterator interface. The executor may not load the whole partition data into memory. > Support query a lot of data. > ---------------------------- > > Key: LIVY-667 > URL: https://issues.apache.org/jira/browse/LIVY-667 > Project: Livy > Issue Type: Bug > Components: Thriftserver > Affects Versions: 0.6.0 > Reporter: runzhiwang > Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time instead of the whole rdd to avoid > OutOfMemory. However, if the largest partition is too big, the OutOfMemory > still occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)