[ https://issues.apache.org/jira/browse/LIVY-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
runzhiwang updated LIVY-667: ---------------------------- Summary: Support query a lot of data. (was: Collect a part of partition to the driver by batch to avoid OOM) > Support query a lot of data. > ---------------------------- > > Key: LIVY-667 > URL: https://issues.apache.org/jira/browse/LIVY-667 > Project: Livy > Issue Type: Bug > Components: Thriftserver > Affects Versions: 0.6.0 > Reporter: runzhiwang > Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When enable livy.server.thrift.incrementalCollect, thrift use toLocalIterator > to load one partition at each time instead of the whole rdd to avoid > OutOfMemory. However, if the largest partition is too big, the OutOfMemory > still occurs. -- This message was sent by Atlassian Jira (v8.3.4#803005)